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'TRAINING  EFFECTIVENESS  EVALUATION  OF 
NAVAL  TRAINING  DEVICES 


ABSTRACT 


This  study  evaluated  the  effectiveness  of  the  Carrier  Air  Traffic 
Control  Center  (CATCC)  training  device  in  training  teams  to  safely 
and  efficiently  control  aircraft  recoveries  and  to  effectively  maintain 
uommunicaticns  necessary  to  implement  this  control  function.  The 
results  indicate  the  device  is  quite  effective  in  that  team,  subteam, 
and  individual  performances  generally  improve  during  training;  that 
student  capability  to  deal  w'th  recovery  contingencies  and  emergencies 
improves;  that  students  generally  perform  acceptably  in  the  opera¬ 
tional  job  setting  and  that  empirical  evidence  supports  (but  does  not 
prove  due  to  the  study  design)  tbe  transfer  of  training  hypothesis; 
and  that  students  gave  the  device  moderate  to  high  ratings  on  realism 
and  effectiveness  characteristics.  It  was  concluded  that  device 
effectiveness  could  be  further  improved  and  that  recommendations 
were  worth  consideration  due  to  the  considerable  impact  CATCC 
lias  on  the  efficiency  and  safety  of  carrier  recovery  operations. 
Recommendations  were  made  relative  to  modifications  and  additions 
to  the  device  hardware  and  software;  utilisation  of  the  device  for 
team  training;  personnel  qualification  requirements,  including 
instructor,  trainer  ’’pilot",  and  student  personnel;  development  of 
CATCC  performance  standards;  and  investigations  which  would 
determine  recommended  training  schedules,  analyze  and  evaluate 
other  portions  of  the  naval  controller  training  program,  and  optimize 
the  controller -display  interface  and  the  CATC  system  with  respect  to 
Naval  Tactical  Data  System  (NTDS)  capabilities. 
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13  ABSTRACT 

This  study  evaluated  the  effectiveness  of  the  Carrier  Air  Traffic  Control  Center 
(CATCC)  training  device  in  training  teams  to  safely  and  efficiently  control  aircraft 
recoveries  and  to  effectively  maintain  communications  necessary  to  implement  this 
control  function.  The  results  indicate  the  device  is  quite  effective  in  that  team, 
subteam,  and  individual  performances  generally  improve  during  training;  that  student 
capability  to  deal  with  recovery  contingencies  and  emc-x^encies  improves;  that  students 
generally  perform  acceptably  in  the  operational  job  setting  and  that  empirical  evidence] 
supports  (but  does  not  prove  due  to  the  study  design)  the  transfer  of  training  hypo¬ 
thesis:  and  that  students  gave  the  device  moderate  to  high  ratings  on  realism  and  ef¬ 
fectiveness  characteristics.  It  was  concluded  that  device  effectiveness  could  be 
further  improved  and  that  recommendations  were  worth  consideration  due  to  the  consider-] 
able  impact  CATCC  has  on  the  efficiency  and  safety  of  carrier  recovery  operations.  Re¬ 
commendations  were  made  relative  to  modifications  and  additions  to  the  device  hardware 
and  software:  utilization  of  the  device  for  team  training;  personnel  qualification  re¬ 
quirements,  including  instructor,  trainer  "pilot",  and  student  personnel;  development 
of  CATCC  performance  standards:  and  investigations  which  would  determine  recommended 
training  schedules,  analyze  and  evaluate  other  portions  of  the  naval  controller  train¬ 
ing  program,  and  optimize  the  controller-display  interface  and  the  CATCC  system  with 
respect  to  Naval  Tactical  Data  System  (NTDS)  capabilities. 
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FOREWORD 


This  study  is  one  in  a  long  term  program  of  in-house  and  contractual 
studies  designed  to  evaluate  the  effectiveness  of  training  devices.  A 
primary  purpose  of  these  field  evaluations  is  to  determine  if  the 
trainers  meet  the  user's  needs,  and  to  provide  guidance  for  improve¬ 
ment  in' the  use  of  simulators  in  particular  training  situations. 

Carrier  landing  is  the  most  dangerous  routine  activity  of  the  naval 
aviator.  An  awareness  of  this  fact,  accompanied  by  the  precise 
requirements  of  the  task,  make  the  recovery  function  one  of  a  very 
high  level  of  stress  for  the  carrier  air  traffic  control  team.  For  this 
reason,  effective  training  in  this  area  is  of  prime  importance  to  the 
Navy. 

The  present  study  has  investigated  student  performance  in  both  simula¬ 
ted  and  operational  settings.  The  experimental  data  provide  evidence 
of  the  effect  that  team  and  individual  skill  levels  have  on  system 
effectiveness  under  both  conditions. 


This  report  has  beej;  arranged  so  that  a  synopsis  of  the  study  is 
presented  in  the  first  four  sections.  For  readers  who  are  interested 
in  delving  deeper,  the  details  of  the  study  are  included  in  the  appendices. 


Project  Psychologist 
Naval  Training  Device  Center 
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SECTION  I 
INTRODUCTION 


PURPOSE  OF  THE  STUDY 

Essential  to  effective  development  and  management  of  training 
devices  is  evaluation  of  their  effectiveness  in  actual  user  environ¬ 
ments  with  respect  to  defined  training  objectives.  The  purpose  of 
this  study  was  to  conduct  such  an  evaluation  on  the  Carrier  Air 
Traffic  Control  Center  (CATCC)  team  training  device  located  at 
the  Fleet  Anti-Air  Warfare  Training  Center  at  Point  Loma,  San 
Diego,  California.  In  broad  terms,  the  primary  objectives  for 
this  device  are  to  train  CATCC  teams  to  safely  and  efficiently 
control  aircraft  recoveries  under  operational  IFR  conditions  and 
to  effectively  maintain  the  communications  with  pilot  and  ship¬ 
board  personnel  required  to  implement  this  control  function. 

The  evaluation  of  CATCC  trainer  effectiveness  was  based  on 
the  following  considerations:  (I)  The  measured  effect  of  training 
in  the  device  on  student  performance  levels,  (2)  Measurements  and 
evaluations  of  later  performance  in  the  operational  environment, 
and  (3)  Factors  found  to  affect  acceptance  and/or  systematic  and 
effective  utilization  of  this  device.  The  first  two  considerations 
determined  the  present  training  effectiveness  of  the  device,  while 
the  third  consideration  determined  whether  device  effectiveness 
could  be  improved  and  how  this  couJd  be  accomplished. 


STUDY  OBJECTIVES 

The  study  design  and  measurement  sets  were  determined  from 
(1)  The  type  of  training  program  in  use,  (2)  Training  and  opera¬ 
tional  schedules  in  effect  during  the  investigation,  and  (3)  Descrip¬ 
tions  and  function  definitions  for  both  the  trainer  and  the  CATCC 
operations  it  simulates.  With  completion  of  these  efforts,  it  was 
possible  to  translate  the  study  purpose  into  specific  study  objectives; 

a.  Do  students  show  evidence  of  improved  performance 
in  the  training  device  as  a  team?  As  subteams?  As 
individuals  ? 
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b.  Do  students  show  evidence  of  learning  to  contend 
with  recovery  contingencies  and  emergencies  in 
the  training  devices? 

c.  Do  students  perform  acceptably  in  the  operational 
setting  after  completion  of  training  and  does  the 
available  evidence  support  the  transfer  of  training 
hypothesis? 

d.  How  do  student  and  instructor  personnel  evaluate 
the  training  device  with  respect  to  training 
effectiveness  characteristics? 

e.  How  do  student  and  instructor  personnel  evaluate 
the  training  device  with  respect  to  trainer  realism 
characteristics  ? 

$j|C 

GENERAL  BACKGROUND  DESCRIPTION 

The  carrier  air  traffic  control  system  consists  of  several  agencies, 
each  with  specific  control  functions  and  responsibilities  for  coordinating 
with  the  other  agencies.  As  one  of  these  agencies,  CATCC  has  primary 
responsibility  for  aircraft  requiring  positive  center  control  (e.g. ,  under 
IFR  conditions )  within  a  fifty  mile  radius  of  the  ship  for  which  that  ship 
is  either  the  destination  or  point  of  departure.  For  aircraft  operating 
under  other  control  conditions,  CATCC  interacts  with  other  agencies 
for  control  purposes  and/or  monitors  to  ensure  traffic  safety. 

One  of  the  more  difficult  CATCC  activities,  and  the  one  which 
requires  the  most  complete  and  fullest  utilization  of  CATCC  capabilities, 


Transfer  of  training  could  not  be  conclusively  demonstrated  under  the 
permissible  study  design.  Evidences  which  could  be  obtained  were 
reviewed,  however,  to  determine  if  the  hypothesis  was  supported.  See 
pp.  32-33  for  a  more  detailed  discussion. 

*♦ 

Detailed  descriptions  of  the  study  and  its  topic,  CATCC,  are  pro¬ 
vided  in  Appendix  1. 
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is  the  Mode  III  recovery  of  a  scheduled  flight  of  aircraft.  The  num¬ 
ber  of  aircraft  per  scheduled  recovery  commonly  ranges  from  7  to 
18.  The  equipment  and  personnel  configurations  for  CATCC  in  full 
operation  have  been  designed  to  implement  this  activit^and  it  is  this 
situation  which  the  CATCC  training  de\  ~e  simulates. 

CATCC  recovery  of  a  flight  of  aircraft  is  considered  to  be  one  of 
the  more  stressful  air  traffic  control  activities.  The  activity  is 
stressful  due  to  the  task  requirements  for  control  within  quite  close 
position  and  time  tolerances  and  for  management  of  what  can  become 
a.  complex  traffic  pattern  with  many  variables  in  operation.  The  level 
of  stress  is  increased  by  the  awareness  of  the  extreme  costs  in  terms 
of  lives  and  aircraft  that  can  be  incurred  by  failing  to  meet  task 
requirements. 


H 


The  trainir^  <gram  for  CATCC  students  at  FAAWTC  consists 
of  both  classroom  lectures  (requiring  one  week  for  completion)  and 
training  device  practice.  Most  of  the  classroom  program  is  com¬ 
pleted  prior  to  device  utilization.  The  device  can  be  utilized  any¬ 
where  from  a  week  to  several  weeks  depending  on  ship  schedules 
and  crew  requirements.  Student  utilization  of  the  device  as  a  team 
results  in  a  dynamic  simulation  of  a  complete  scheduled  Mode  III 
recovery  and  provides  training  in  both  team  coordination  and  the 
individual  skills  unique  to  each  position.  Such  recoveries  have  a 
basic  standard  form  (imposed  by  the  objective  and  implementation 
methods)  and  can  therefore  be  described  as  a  problem  which  is 
practiced  repetitively  by  student  teams  until  proficiency  is  acquired. 
The  problems  are  made  more  or  less  difficult  by  the  insertion  of 
problem  variables,  i.  c.  ,  certain  contingency  and  emergency  situa¬ 
tions,  by  instructor  personnel. 


i 


1 


A  glossary  is  provided  on  page  17. 

** 

A  variation  on  this  activity,  the  random  Mode  III  recovery,  makes 
very  similar  requirements  on  CATCC  personnel  and  is  occasionally 
simulated  in  the  training  device.  However,  due  to  its  similarity  and 
infrequent  presentation  during  training,  it  will  not  be  discussed 
further  here. 
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Students  are  ship-based  and  come  as  a  group  from  that  ship  to 
the  trainer.  I.  e.  ,  these  students  are  preparing  to  work  together 
upon  deployment  rather  than  preparing  as  individuals  who  will  later 
receive  separate  assignments.  Further,  whatever  their  previous 
experiences  and  training,  the  training  received  in  this  device 
represents  the  final  preparation  prior  to  deployment.  The  next  use 
of  their  control  capabilities  is  in  the  actual  operational  environment 
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SECTION  II 


SUMMARY  OF  STUDY  RESULTS 


The  results  ot  the  study  will  be  summarized  by  answering  the 
questions  posed  as  study  objectives.  A  detailed  discussion 

of  results  and  suggestions  appears  in  Appendix  B. 

a.  Do  students  show  evidence  of  improved  performance 
in  the  CATCC  training  device  as: 

Teams?  Yes,  team  performance  clearly  improves 
as  a  result  of  device  utilization  (see  detailed 
results  for  the  objective  performance  measures 
of  aircraft  separation  error  and  recovery  dura¬ 
tion,  pp.  43-52). 

Subteams?  Yes,  subteam  performance  does 
improve  as  a  result  of  device  utilization  (see 
objective  performance  measures  of  azimuth 
and  glideslope  errors  and  of  communications 
efficiency  and  noise  level,  pp.  53-64). 

Individuals?  Yes,  individual  performance  does 
generally  improve  or  is  corrected  as  a  result 
of  device  training  (pp.  64-72  ).  The  two  excep¬ 
tions  appear  to  result  from  trainer  utilization 
rather  than  trainer  capability  (pp.  67-69,  70-72  ). 

b.  Do  students  show  evidence  of  learning  to  contend  with 
recovery  contingencies  and  emergencies  in  the  train¬ 
ing  device?  Yes.  Not  only  do  they  develop  approaches 
to  these  problems  but  performance  on  objective  meas¬ 
ures  indicates  an  increasing  capability  to  meet  system 
criteria  during  difficult,  as  well  as  easy,  recoveries 
(pp.  44-64). 

c.  Do  students  perform  acceptably  in  the  operational 
setting  after  completion  of  training  and  does  the 
available  evidence  support  the  transfer  of  training 
hypothesis?  The  objective  performance  data  indicate 
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the  answers  are  yes.  The  students  generally  per¬ 
formed  acceptably  with  better  operational  perform¬ 
ances  given  by  those  who  received  more  training 
(pp.  43-64,  bo-67).  The  evaluative  performance 
data  (judgments  made  by  the  controllers,  pilots, 

LSOs,  and  ORI  team)  all  indicate  acceptable  opera¬ 
tional  performances  after  completion  of  training 
(pp.  7  3-75). 

d.  How  do  student  and  instructor  personnel  evaluate  * 

the  training  device  with  respect  to  training  effective¬ 
ness  characteristics?  Both  personnel  groups  rated 

the  trainer  as  moderately  to  highly  effective  across 
all  the  effectiveness  characteristics  (pp.  75-81), 

Critical  comments  and  suggestions  to  improve  trainer 
effectiveness  are  discussed  (pp.  84  -86). 

e.  How  do  student  and  instructor  personnel  evaluate  the 
training  device  with  respect  to  trainer  realism 
characteristics?  Two  sets  of  realism  characteristics 
were  evaluated,  task  environment  realism  and  task 

performance  realism.  Both  personnel  groups  rated  j 

the  trainer  as  moderately  to  highly  realistic  on  both 

sets  of  '•ealism  characteristics  (pp.  75-  81  ).  Critical 

comments  and  suggestions  to  improve  trainer  realism 

are  presented  (pp.  81  -  84). 

Comparison  of  ratings  given  to  trainer  realism  vs.,  training  effective¬ 
ness  characteristics  indicates  the  device  generally  received  higher  ratings 
on  effectiveness  than  on  realism.  When  students  were  asked  to  reevaluate 
the  training  device  after  having  completed  their  first  operational  recoveries 
of  the  type  for  which  they  had  received  training,  they  rated  the  device  as 
less  real,  but  as  more  effective  than  they  had  while  in  training  (pp.  75-81), 
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SECTION  HI 
CONCLUSIONS 

The  CATCC  trainjjig  device  has  been  evaluated  from  three  stand¬ 
points:  (1)  Evaluation  of  the  effect  of  repeated  performance  in  the 
trainer  on  team,  subte#.rn  and  individual  performance  levels  during 
the  training  period,  (2)  Evaluation  of  the  acceptability  of  team,  sub¬ 
team  and  individual  ptjTorrnance  levels  during  initial  recovery  opera¬ 
tions  at  sea,  and  (3)  Evaluation  of  the  training  device  in  terms  of 
specific  effectiveness  g.nd  realism  characteristics.  The  conclusions 
regarding  each  of  fhe$o  are  discussed  in  the  following  paragraphs. 

EFFECT  OF  REPEATED  PERFORMANCE  IN  THE  TRAINER 

Because  the  majority  of  objective  measures  of  performance  used 
in  this  study  are  related  to  system  effectiveness  criteria  for  the  CATC 
system,  a  demonstration  of  improvement  in  measured  performance 
during  training  is  especially  relevant  to  carrier  recovery  operations 
involving  CATCC.  Da ta  collected  on  these  objective  performance 
measures  in  the  training  device  indicate  the  device  is  a  very  effective 
team  and  subteam  traii^m  and  can  be  effective  as  an  individual  trainer 
when  necessary.  The  data  further  indicate  that  each  team  and  sub¬ 
team  composition  tencl^  to  he  a  somewhat  unique  entity  which  benefits 
from  training  as  an  equity.  Although  it  must  be  assumed  that  some 
aspects  of  previous  controller  training  and  experience  transfer  to  a 
new  team  or  subteam  c°*nposition,  empirical  evidence  suggests  a 
definite  need  for  and  benefit  from  training  by  newly  formed  teams 
and  subteams. 

ACCEPTABILITY  OF  OPERATIONAL  PERFORMANCE 

Operational  performance,  as  measured  by  the  objective  perform¬ 
ance  measures,  was  fo^d  to  be  acceptable.  Further,  evaluations  of 
CATCC  performance  by  the  controllers  themselves,  pilots,  LSOs,  and 
the  ORI  team  again  indicate  that  operational  performance  was  acceptable. 

The  differences  in  measured  and  observed  performances  between  teams, 
subteams  and  individuals  support  the  transfer  of  training  hypothesis  in  that 
personnel  with  more  twining  performed  better.  I.  e.  ,  use  of  the  CATCC 
trainer  for  training  purposes  appears  to  positively  impact  on  operational 
Mode  III  scheduled  recoveries.  The  evidence  is  not  conclusive,  however, 
because  the  study  could  not  he  structured  so  as  to  control  or  positively 
account  for  possible  confounding  variables. 
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TRAINING  DEVICE)  CHARACTERISTICS 

The  training  device  was  evaluated  as  being  moderately  to  highly 
training  effective  and  realistic  in  terms  of  task  environment  and 
performance.  Evaluations  of  controller  performance  by  both  con¬ 
trollers  and  pilots  and  evaluations  of  device  characteristics  indicate 
that,  while  the  device  is  quite  training  effective  with  respect  to  con¬ 
tingency  and  emergency  situations,  it  is  not  quite  as  effective  with 
respect  to  guidance  training.  (Suggestions  for  improving  trainer 
effectiveness  were  made  with  respect  to  several  device  training  and 
realism  characteristics,  including  guidance  training  (pp.  73-86).  ) 
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SECTION  IV 


RECOMMENDATIONS 


B- 


Several  recommendations  can  be  made  based  on  the  results  of 
this  investigation  which,  directly  or  indirectly,  relate  to  CATCC 
effectiveness.  These  recommendations  fall  into  the  categories  of: 

(1)  Capabilities  and  utilization  of  the  CATCC  training  device,  (2) 
Considerations  regarding  the  rest  of  the  controller  training  program, 
and  (3)  Considerations  regarding  the  CATCC  man-machine  system. 

C^TCC  TRAINING  DEVICE  RECOMMENDATIONS 

Specific  detailed  discussions  and  suggestions  relating  to  train¬ 
ing  device  modifications  and  utilization  appear  throughout  Appendices 
1  and,  especially,  ?.  and  the  reader  is  encouraged  to  review  these 
sections.  Only  a  few  of  these  suggestions  will  be  replicated  here. 

Other  recommendations  presented  here  represent  a  synthesis  of  the 
detailed  discussions  with  observations  and  interviews  conducted 
throughout  the  investigation.  The  recommendations  presented  below 
fall  into  five  areas:  (1)  Hardware  and  software  considerations,  (2) 
Utilization,  (3)  Personnel,  (4)  Performance  standards,  and  (5)  Train¬ 
ing  schedules. 

HARDWARE  AND  SOFTWARE  CONSIDERATIONS  To  the  extent  that 
training  effectiveness  is  determined  by  trainer  acceptance,  training 
effectiveness  will  be  improved  (and  capabilities  expanded)  by  more 
realistic,  complete,  and  reliable  simulations  of  specific  CATCC  con¬ 
figurations,  problems,  and  operating  environments.  For  example, 
the  addition  of  ICS  and  Bolter/ Waveoff  position  simulation  capabilities 
by  addition  of  the  appropriate  hardware  and  hardware/ software  inter¬ 
faces  is  particularly  recommended  because  of  the  impact  these  equip¬ 
ments  can  have  on  team  coordination  procedures.  It  was  found  that 
the  crew  from  one  carrier  did  not  completely  accept  the  device  as  a 
training  instrument  due  to  its  inability  to  simulate  these  aspects  of 
their  specific  operating  environment.  Further,  the  evidence  suggested 
that  operational  events  which  could  have  been  very  costly  might  have 
been  avoided  if  the  crew  members  had  been  able  to  train  with  their 
Bolter/ Waveoff  position  simulated  (discussed  on  pp.  83-85). 
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The  capability  of  the  device  to  simulate  specific  CATCCs  could 
be  further  expanded  by  the  addition  of  the  other  NTDS  radar  equip¬ 
ments  in  use  and  the  capability  to  configure  these  equipments  for 
training  purposes.  The  extent  to  which  this  would  be  desirable 
should  be  determined  from  a  careful  analysis  of  feasibilities, 
alternatives,  costs,  and  resultant  improvements  in  training  device 
flexibility  and  effectiveness. 

The  addition  of  a  specific  hardware  item,  the  second  channel 
of  the  SPN  42  PAR,  is  not  only  recommended  -  it  is  expected  to 
become  a  reality  in  1972.  At  present  only  one  SPN  42  is  in  the 
training  device  and,  as  a  result.  Final  Controller  training  for  SPN 
42-equipped  carriers  is  not  efficient  and,  probably,  not  entirely 
effective. 

Based  on  comments  made  by  students  and  instructors,  device 
training  is  presently  incomplete  in  that  the  Selective  Identification 
Feature  (SIF)  of  the  IFF  equipment  cannot  presently  be  simulated. 
This  feature  could  reasonably  be  implemented  with  hardware  and/or 
software  modification.  Since  the  CATCC  device  represents  the  final 
opportunity  in  training  for  students  to  integrate  skills  and  capabilities 
into  coordinated  activity,  and  SIF  represents  a  capability  used  in  the 
operational  environment,  this  recommendation  should  be  given  con¬ 
sideration. 

The  major  difficulties  and  frustrations  of  the  CATCC  training 
device, from  the  standpoint  of  both  instructors  and  students,  can  be 
summed  up  intwo  words:  software  problems.  As  discussed  else¬ 
where  (p.  31),  thirty  five  percent  of  the  observed  recovery  problems 
were  terminated  prior  to  completion  due  to  problems  in  the  software 
system.  Several  other  recoveries  were  completed  with  difficulty. 
Training  time  was  further  reduced  as  a  result  of  software  system 
failures  which  rendered  the  trainer  inoperable  for  periods  of  time. 
Simulation  realism  was  degraded  and  controller  irritation  caused 
by  inappropriate  target  behaviors  resulting  from  programming 
inadequacies  and  system  problems  (see  pp.  81  and  82  for  examples). 
Further,  attempts  to  realize  a  workable  program  to  automate  the 
NTDS  symbology  tracking  function  have  thus  far  been  frustrated 
(tracking  is  currently  performed  manually  in  the  device  and  thus 
represents  an  additional  task  which  realizes  little  benefit  for  the 
student  controller). 
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The  combined  effects  of  the  above  problems  is  incomplete  use 
of  training  time  and  an  irritation  which  reduces  student  acceptance, 
frustrates  instructional  functions  of  the  device,  and  results  in  less 
than  the  possible  device  training  effectiveness  and  efficiency.  It  is 
reccmmended"that  consideration  be  given  to  the  following:  (1)  Pro- 
t  vision  of  increased  support  to  the  programming  efforts  underway  both 

for  CATCC  and  for  the  overall  training  system  at  FAAWTC;  (2)  The 
possibility  of  redesigning  the  software  system  with  respect  to  CA.TCC. 
This  device  can  operate  independently  from,  rather  than  in  coordina¬ 
tion  with,  the  other  training  devices  constituting  the  overall  training 
system.  As  a  result,  a  redesign  of  the  software  and  computer  hard¬ 
ware  systems  might  benefit  both  CATCC  and  other  device  operations; 

(3)  An  investigation  to  identify  the  downtime  and  degraded  operation 
causes, and  to  determine  alternative  approaches  to  resolve  identified 
software  and  computer  hardware  problems  for  both  CATCC  and  the 
overall  training  system  at  FAAWTC;  and  (4)  Selection  and  implementa¬ 
tion  of  alternatives  idertified  in  the  investigation. 

UTILIZATION  At  present  the  CATCC  training  device  is  utilized  for 
several  purposes  other  than  team  training  (see  pp.  30-  31 ).  In  train¬ 
ing  periods  observed  during  this  investigation,  it  was  extensively 
used  to  cross-train  individual  controllers  and  to  select  the  better 
team  and  subteam  compositions.  Use  of  the  trainer  as  a  team  and 
subteam  selection  device  indicates  a  high  evaluation  of  the  validity 
of  trainer  performances  on  the  part  of  CATCC  supervisory  personnel 
and,  based  on  the  results  of  this  investigation,  that  use  of  the  trainer 
cannot  be  faulted.  Use  of  the  CATCC  training  device  to  cross-train 
individuals  can  be  faulted,  however,  to  the  extent  that  it  interferes 
with  training  of  the  selected  teams  and  subteams.  As  observed  pre¬ 
viously,  each  team  appears  to  represent  a  unique  entity;  an  entity 
which  requires  and  benefits  from  device  training.  When  cross-training 
is  done  to  the  point  that  the  teams  selected  for  operational  duty  receive 
little  training  in  their  own  unique  team  configurations,  then  the  trainer 
is  not  being  used  effectively.  Especially  inasmuch  as  the  team  training 
’  environment  is  not  one  where  a  great  deal  of  individualized  instruction 

and  attention  can  be  given  without  interfering  with  team  training  require¬ 
ments.  I.  e. ,  neither  team  nor  individual  training  is  optimized  in  the 
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cross-training  situation.  The  CATCC  trainer  admittedly  is  c  onvenient 
for  cross-training  but  it  is  recommended  that  it  be  used  for  such  only 
to  the  extent  that  it  is  necessary  to  ensure  position  coverage  and  that 
sufficient  training  time  will  still  be  available  for  selected  team  train¬ 
ing. 

PERSONNEL  In  some  discussions  of  trainer  effectiveness,  a  distinc¬ 
tion  has  been  made  between  the  trainer's  training  capability  and  its 
utilization.  It  must  be  pointed  out  that  whatever  a  trainer's  potential 
for  training  may  be,  it  will  not  be  fully  realized  unless  it  is  utilized 
properly.  It  is  sometimes  the  case  that  a  rather  inadequate  training 
device  is  made  highly  effective  by  the  manner  in  which  it  is  utilized 
by  instructor  personnel  to  implement  the  training  program.  Whatever 
the  capabilities  of  the  CATCC  training  device  may  be,  it  has  been  made 
highly  effective  by  the  manner  in  which  its  instructor  personnel  have 
used  it.  The  value  of  highly  capable  instructors  in  this  training  device 
environment  is  perhaps  best  explicated  by  student  responses  in  the 
Trainer  Evaluation  form:  "Instructors  are  highly  skilled  in  problems 
for  CCA"  and  "In  training  you  have  a  chance  to  stop  things  and  discuss 
the  problems  as  they  are  happening,  and  discuss  them  with  highly 
qualified  instructors.  "  It  is  recommended  that  the  present  level  of 
instructor  capability  be  retained  to  implement  the  CATCC  training 
device  programs.  The  effectiveness  of  this  device  appears  to  be  very 
much  a  function  of  its  utilization  by  instructor  personnel;  if  this  is 
indeed  the  case,  then  its  continued  effectiveness  rests  with  the  con¬ 
tinuance  of  a  highly  qualified  instructor  staff. 

At  the  present  time  there  are  no  stated  specific  qualification 
requirements  for  CATCC  instructor  personnel;  the  level  of  instructor 
capability  and  motivation  has  been  maintained  as  a  result  of  timely 
action  by  the  CATCC  training  staff.  It  is  recommended  that  personnel 
qualifications  be  identified  and  documented  for  this  billet  such  that 
the  desired  level  of  capability  and  motivation  will  be  maintained. 

Another  personnel  factor  affecting  trainer  effectiveness  is  the 
PC&E  "pilot",  who  plays  an  important  role  in  the  implementation  of 
recovery  problems.  The  discussion  of  this  factor  is  presented  else¬ 
where  (p.  82  )  and  will  not  be  reiterated  here.  Due  to  the  impact 

these  personnel  can  have  on  recovery  simulation  adequacy,  it  is 
recommended  that  they  be  permanently  assigned  to  the  CATCC  device 
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and  have,  to  some  extent  at  least,  an  aviation  background.  Giv^n 
personnel  meeting  these  requirements,  the  CATCC  instructor  staff 
could  maintain  the  level  of  "pilot"  performance  required  to  provide 
a  realistic  simulation. 

A  final  personnel  factor  that  warrants  consideration  is  that  of 
the  student.  The  effectiveness  of  the  CATCC  training  device  is 
constrained  to  the  extent  that  (1)  Members  of  a  crew  are  at  dis¬ 
similar  levels  with  respect  to  training  and  capability,  with  some 
members  not  yet  ready  for  team  training,  and  (2)  Incomplete  teams 
are  sent  from  the  ship.  The  first  item  is  a  constraint  because  it  tends 
to  reduce  the  possible  overall  difficulty  level  of  the  recovery  problems 
(thus  limiting  the  amount  of  proficiency  and  knowledge  that  can  be 
developed)  and  to  present  conflicting  instructional  requirements  that 
can  be  difficult  to  resolve  during  recovery  problem  development. 

The  second  item  is  a  serious  constraint  both  in  terms  of  the  team 
training  requirements  discussed  previously  and  because  the  realism 
of  the  simulation  is  degraded.  It  is  recommended  that  the  basis  for 
CATCC  assignments  be  reviewed  and  that  CATCC  crew  members  be 
more  systematically  scheduled  for  device  training. 

PERFORMANCE  STANDARDS  Inasmuch  as  the  training  device  has  been 
found  learning  effective  and  appears  to  affect  operational  CATC  system 
performance  levels,  it  appears  reasonable  to  suggest  that  CATC  system 
operations  involving  CATCC  would  benefit  from  the  development  and 
implementation  of  CATCC  personnel  qualification  requirements  using  the 
CATCC  trainer  as  the  testing  instrument.  If  the  training  device  if 
capable  of  effectively  serving  operatioral  needs,  and  it  appears  it  is, 
then  the  necessary  knowledges  and  implementation  structure  should  be 
developed  to  optimize  its  impact  on  recovery  operations.  The  establish¬ 
ment  of  (1)  Standardized  skill  levels  for  those  aspects  of  operational 
performance  which  can  be  predicted  from  trainer  performance  and  (2) 
Requirements  that  these  levels  be  met  prior  to  deployment,  would  avoid 
inefficiencies  and  safety  hazards  resulting  from  CATCC  performance  (of.  , 
pp.  48-52  and  66-67  ).  It  is  therefore  recommended  that  a  research 
program  be  conducted  to  determine  those  qualification  requirements 
valid  for  operational  performance  and  that  the  procedures  and  regula¬ 
tions  necessary  to  implement  these  requirements  be  developed. 

TRAINING  SCHEDULES  It  is  not  presently  known  what  amount  and 
scheduling  of  CATCC  device  training  would  best  serve  the  needs  of 
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the  various  types  of  crews  using  the  device.  In  the  present  study,  for 
instance,  it  was  felt  subjectively  that  the  crew  from  one  carrier  had 
not  received  enough  framing,  while  the  crew  from  the  other  carrier 
had  perhaps  been  overtrained.  It  is  suggested  that  an  investigation 
be  conducted  to  determine:  (1)  What  levels  of  trainer  performance 
provide  the  best  transfer  to  the  operational  environment,  (2)  What 
factors  other  than  performance  levels  determine  optimum  training 
schedules,  and  (3)  Whether  or  not  a  set  of  recommended  training 
schedules  can  be  developed  for  the  various  types  of  crews. 


CONTROLLER  TRAINING  PROGRAM  RECOMMENDATIONS 


The  CATCC  training  device  at  FAAWTC  is  one  of  the  devices 
providing  team  training.  There  are  two  other  devices  for  team 
training  (at  Glynco  and  Norfolk)  and  there  are  various  preparatory 
routes  to  these  devices.  It  is  suggested  that  these  other  programs 
may  benefit  from  effectiveness  evaluations  similar  to  the  one  reported 
here.  It  is  also  suggested  that  an  analysis  of  the  overall  naval  con¬ 
troller  training  program  in  terms  of  operational  requirements,  inter¬ 
face  requirements  between  discrete  program  units,  and  student  input 
characteristics  would  result  in  a  more  efficient  program  which  would 
more  effectively  serve  the  many  different  types  and  levels  of  opera¬ 
tional  requirements  for  controller  personnel. 


CATCC  MAN-MACHINE  SYSTEM  RECOMMENDATIONS 


The  introduction  of  NTDS  impacts  on  carrier  operations  in  a 
number  of  different  ways,  including  capability  expansions,  alternative 
procedural  possibilities,  and  man-machine  interface  modifications. 
How  to  effectively  integrate  NTDS  into  carrier  operations  so  that  it 
will  best  serve  operational  objectives  and  requirements  cannot  be 
said  to  be  completely  resolved  yet,  at  least  in  the  case  of  CATCC. 

At  least  two  investigations  are  recommended  which  relate  to  the 
integration  of  NTDS  into  CATCC  operations. 


T.  will  be  recalled  that  performance  trends  on  one  measure  of 
team  performance  (Aircraft  Separation  Error,  p.  48),  crew  rejection 
of  NTDS  symbology  after  a  confliction  during  training  (Number  of 
Conflictions,  p.  66),  and  controller  evaluations  of  NTDS  symbology 
effectiveness  (p.  85),  all  raised  the  question  of  whether  or  not  NTDS 
symbology  should  be  provided  in  CATCC  and/or  how  it  should  be 
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provided.  It  is  therefoie  recommended  that  an  investigation  be  con¬ 
ducted  to  determine  if  and  how  the  controller  -display  interface  should 
he  modified  with  respect  to  the  NTDS. 

Secondly,  it  is  recommended  that  detailed  consideration  be  given 
on  how  to  integrate  NTDS  capabilities  and  CATCC  capabilities  to 
realize  their  full  combined  potential  and  to  maintain  or  improve  CATC 
system  performance.  The  development  of  NTDS  allows  for  considera¬ 
tion  of  alternative  possibilities  (e.  g. ,  computer  controlled  approaches 
and  landings  using  the  Automatic  Carrier  Landing  System).  Tradeoffs 
are  required  with  respect  to  capability  values  and  the  requirements  to 
maintain  these  capah  lities.  Thus  far,  considerable  effort  has  been 
devoted  to  the  machine  portion  of  the  man-machine  systems  affected 
by  the  introduction  of  NTDS.  It  is  suggested  that  increased  attention 
be  given  to  the  man  (operator)  portion  of  these  systems,  and,  as  an 
important  man-machine  system,  CATCC  warrants  this  additional  con¬ 
sideration. 
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APPENDIX  A 
STUDY  DESCRIPTION 


SUBJECT  POPULATION 

Data  were  collected  on  CATCC  crews  from  two  carriers, 
described  herein  as  Ships  One  and  Two.  A  total  of  23  CATCC 
personnel  were  observed  in  the  trainer;  subsequently,  15  of  these 
were  observed  in  the  operational  environment  while  performing 
their  first  scheduled  Mode  III  recoveries.  (A  more  detailed  data 
description  appears  ir.  Table  1.  )  These  recoveries  took  place  two 
months  after  the  final  training  period  and  were  preceded  by  on¬ 
board  CATCC  activities  that  can  best  be  described  as  part-task 
on-the-job  training  for  scheduled  Mode  III  recoveries. 

The  personnel  from  Ship  One  were  observed  during  their  first 
and  third  (the  final)  weeks  of  device  training.  Ship  Two  personnel 
were  observed  during  their  fifth  and  final  week.  It  should  be  noted, 
however,  that  due  to  changes  made  in  crew  compositions  and  posi¬ 
tion  assignments  during  training  that  the  total  training  received  by 
any  one  individual,  subteam,  or  team  may  be  substantially  less 
than  3  to  5  weeks.  For  example,  data  collected  during  Ship  One's 
third  week  of  training  described  only  first  or  second  week  perform¬ 
ances.  Data  for  Ship  Two  will  be  discussed  as  representing  the 
fifth  week  of  training  but,  in  fact,  it  represents  something  less  than 
this.  It  should  also  be  noted  that  the  week-long  device  training 
periods  were  generally  separated  by  a  one  to  two  month  time  interval. 

In  view  of  the  impact  that  team,  subteam,  and  individual  identity 
was  found  to  exert  on  performance  level,  a  detailed  description  of 
team  identities  in  the  training  device  as  opposed  to  those  on  board 
the  carriers  is  v/arranted.  These  differences  will  be  recalled  later 
ir  discussions  of  Results  as  they  relate  to  questions  of  transfer  of 
training  and  correlation  of  trainer  and  ship  performance  levels. 

Two  team  compositions  were  observed  on  board  Ship  One,  neither 
of  which  was  exactly  like  any  of  the  team  compositions  measured  in 
the  trainer.  The  first  team  observed  contained  two  personnel  who 
had  not  received  training  in  the  trainer  (identified  as  Team  Nine  in 
Results  Section).  Unfortunately,  due  to  the  constraints  imposed  by 
coordination  requirements  and  too  few  data  collection  personnel, 
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TABLE  1.  STUDY  DATA  DESCRIPTION 


DATA  DESCRIPTOR 

OBSERVED  JN 
TRAINING  DEVICE 

OBSERVED  AT  SEA 

SHIP  ONE 

SHIP  TWO 

SHIP  ONE 

SHIP  TWO 

Performance  Period-., 

1st  &  3rd 

5th  Week* 

2  days 

2  days 

We.-ks* 

Number  of  Student 

Personnel (excluding 

CCAO  Xi  boardkeepers) 

14 

9 

9 

6 

Number  of  Students 

Who  Performed  as: 

Marshal 

8 

2 

1 

1 

Approach 

8 

5 

3 

2 

Final 

11 

6 

4 

2 

Supervisor 

5 

4 

2 

1 

Number  of  Recoveries 

32  ** 

14  ** 

4 

4 

Number  of  Teams 

Performing  More  Than 

One  Recovery 

9 

2 

2 

1 

Number  of  Subteams 

Performing  More 

Than  One  Recovery 

17 

8 

5 

2 

*  The  third  (Ship  One)  and  fifth  (Ship  Two)  weeks  were  the  final 

training  periods  followed  m  both  cases  by  operational  recoveries 
of  the  same  type  two  months  later.  Operational  data  were  collected 
at  these  times. 

**  Not  all  of  the  observed  recoveries  were  completed  due  to  software 
problems.  Depending  on  extent  of  recovery  completion,  data  was 
obtained  on  some  subset  of  performance  measures. 
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data  could  not  be  collected  on  several  measures  for  this  first  team. 

In  the  second  team  observed  on  Ship  One  (Team  Ten),  the  two 
"untrained"  personnel  had  been  replaced  by  persons  who  had  per¬ 
formed  many  recoveries  in  the  trainer  in  their  assigned  positions. 

In  addition,  one  of  the  "trained"  personnel  on  the  first  team  had 
been  replaced  by  a  controller  with  considerable  previous  experience 
in  that  position  but  who  had  not  practiced  in  that  position  as  a  team 
member  at  any  time  during  the  training  period.  Most  of  the  data 
presented  will  describe  performance  by  this  second  team.  Where 
extrapolations  from  trainer  data  to  onboard  data  for  Ship  One 
appeared  reasonable  due  to  judged  or  exact  similarities  of  team, 
subteam,  or  individual  identify  in  trainer  and  ship  environments, 
this  will  be  discussed.  Where  "trained"  vs.  "nontrained"  perform¬ 
ance  comparisons  appear  possible,  these  will  also  be  pointed  out. 

The  team  composition  for  Ship  Two  data  in  the  trainer  vs.  on 
board  the  ship  was  not  identical,  but  was  considered  to  be  quite 
similar.  One  of  the  Approach  control  positions  on  board  the 
carrier  was  filled  by  a  controller  who  had  received  a  week  of 
training  in  the  trainer  with  the  other  team  members  during  their 
first  week  of  device  training.  Again,  trainer  vs.  ship  extrapola¬ 
tions  and  comparisons  based  on  performance  data  will  be  based 
on  overall  team  and  subteam  similarities,  and  the  consideration 
that  one  individual  member  was  evaluated  as  talented,  but  had  also 
received  less  training. 

CATCC  and  CATCC  OPERATIONS 

The  primary  control  positions  for  Mode  III  recovery  operations 
on  these  ships  are:  Marshal  and  Subteams  A  and  B,  where  each  sub¬ 
team  consists  of  an  Approach,  or  Letdown,  Controller  and  a  Final 
Controller.  Other  personnel  directly  involved  in  support  or  super¬ 
visory  roles  include  the  Carrier  Controlled  Approach  Officer  (CCAO), 
the  Supervisor,  and  Boardkeepers  for  the  marshal  and  final  status 
boards.  These  positions  are  described  in  greater  detail  below.  Each 
of  the  performance  measures  used  in  this  study  (listed  in  Tables  3  and 
4)  measured  one  of  the  following:  (1)  The  entire  team,  with  emphasis 
on  the  primary  positions;  (2)  Each  of  the  subteams;  and  (3)  Individual 
controller  positions.  The  basic  personnel  configuration  for  both  the 
trainer  and  Ships  One  and  Two  is  described  in  Figure  1. 
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During  recovery  operations,  the  aircraft  are  initially  under 
Marshal  control.  The  Marshaller  organizes  aircraft  within  the 
marshalling  configuration  and  ensures  their  individual  entry  into 
the  approach  pattern  at  the  appropriate  time.  Aircraft  handoffs 
from  Marshal  to  Approach  alternate  between  Subteams  A  and  B. 

The  handoff  points  on  a  recovery  profile  between  Marshal  and 
Approach  Controller  A  or  B,  and  then  between  Approach  and 
Final  Controllers  are  indicated  in  Figure  2.  (Although  there  are 
several  possible  recovery  profiles.  Figure  2  describes  the  one 
generally  used  in  the  trainer  and  observed  during  operational 
recoveries.  )  Approach  and  Final  Controllers  are  responsible 
for  keeping  aircraft  aligned  with  the  approach  profile  and  path. 
Approach  Controllers  are  additionally  responsible  for  maintain¬ 
ing  adequate  separation  between  aircraft.  If  the  aircraft  enters 
a  go-around  (G/A),  waveoff  (W/O),  or  bolter  status,  the  pilot 
then  enters  the  bolter-waveoff  pattern  which  parallels  the  approach 
path  in  the  opposing  direction.  The  Approach  Controller  then 
assumes  his  other  function,  which  is  to  integrate  that  aircraft 
back  into  the  approach  path. 

The  flow  for  CATCC  control  and  integration  functions  during 
scheduled  Mode  III  recoveries  is  presented  in  Figure  3.  The 
information  flow  which  supports  and  implements  these  functions 
is  described  in  Figure  4. 

In  addition  to  the  brief  description  of  position  responsibilities 
provided  by  the  above  paragraphs,  the  following  details  are 
pertinent: 

CCAO  AND  SUPERVISOR  -  these  persons  are  responsible  for  the 
CATCC  team  and  their  actions.  The  CCAO  provides  the  major 
interface  between  CATCC  and  other  ship  agencies  which  provide 
essential  information  and  require  coordination.  The  Supervisor 
coordinates  the  activities  within  CATCC  as  needed  and  is  at  the 
immediate  command  of  any  controller  in  the  center  for  information 
or  direction. 

MARSHALLER  -  as  the  initial  contact  and  controller  for  aircraft 
destined  for  the  ship,  he  provides  essential  information  (e.g. , 
weather,  altimeter  setting,  bearing,  assigned  airspace),  collects 
essential  information  (e.g.,  position,  fuel  state,  problems  affecting 
recovery),  and  is  responsible  for  the  timing  and  sequence  of 
departures  from  the  marshall  area. 
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Figure  3.  Carrier  Mode  III  Scheduled  Recoveries:  Air  Traffic 
Control  Function  Flow 
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APPROACH  CONTROLLER  (A  OR  B)  -  the  approach  controllers, 
like  the  marshaller,  use  area  surveillance  radar  (ASR)  scopes  to 
perform  their  control  and  integration  functions.  And  it  is  these 
scopes  to  which  Naval  Tactical  Data  System  (NTDS)  symbology  is 
being  introduced;  these  are  radar  symbols  which  track  assigned 
.  aircraft  and  can  be  used  by  the  controller  to  obtain  supplementary 

information  such  as  altitude. 

FINAL  CONTROLLER  (A  OR  B)  -  the  finwl  controllers  monitor 
precision  approach  radar  (PAR)  screens,  closely  controlling  air¬ 
craft  with  respect  to  glideslope  and  azimuth  position  until  either 
PAR  minimums  are  reached  or  the  LSO  takes  over. 

STATUS  BOARD  KEEPERS  (Marshal  and  Final)  -  these  men  are 
responsible  for  properly  displaying  and  updating  flight  and  air¬ 
craft  information  on  their  respective  boards. 


THE  CATCC  TRAINING  DEVICE  J 

il 

•3 

DEVICE  DESCRIPTION  The  CATCC  Trainer  consists  of  two  f 

rooms.  One  room  is  a  CATCC  mockup  utilizing  actual  operating  | 

equipments  to  implement  control  positions.  In  operation,  the  | 

mockup  provides  a  complete  and  realistic  environment  for  team  | 

control  of  a  scheduled  recovery.  I 


The  other  room,  called  the  CATCC  Program  Control  and 
Evaluation  (PC&E)  center,  contains  the  software  interface  and 
display  facilities  necessary  to  simulate  video  displays  of  air¬ 
craft  and  radar  (ship  and  NTDS)  symbology  on  the  screens 
located  in  the  mockup,  to  monitor  student  performance,  and  to 
insert  problem  variables.  The  aircraft  movement  patterns 
represented  by  the  displayed  video  are  determined  by  control 
actions  at  pilot  stations  in  PC&E  made  in  response  to  student 
controller  commands.  These  patterns  are  implemented  by 
computer  software.  A  small  section  located  in  the  mockup 
behind  the  primary  area  of  team  activity  provides  mockup 
instructor  personnel  with  communication  lines  to  PC&E  personnel 
which  are  separate  from  the  control  voice  channels  between  the 
students  and  PC&E  "pilots". 
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Although  all  CVA  CATCCs  serve  the  same  control  functions, 
coordinate  with  the  same  ship  agencies,  and  implement  these 
functions  with  radar,  voice  channels  and  status  boards,  no  two 
CATCCs  are  exactly  alike.  They  differ  in  equipments  (e.  g. ,  SPN 
10  vs.  SPN  35  vs.  SPN  42  PARs,  ICS  available  vs.  no  ICS  avail¬ 
able,  etc.  ),  equipment  arrangement  (although  the  general  configura¬ 
tion  is  as  described  in  Figure  1),  coordination  procedures  within 
CATCC  and  between  CATCC  and  other  agencies,  ship  procedures 
and  doctrine,  problems  encountered,  and  manner  of  implementing 
recoveries  (e.  g. ,  recovery  profile  utilized,  handoff  points  on  the 
profile  between  positions,  specialized  personnel  assignments  such 
as  maintenance  of  special  displays,  etc.  ). 

The  CATCC  trainer  has  been  assembled  to  simulate  a  generalized 
CATCC  with  respect  to  positions,  position  configuration,  and  equip¬ 
ment.  The  trainer  has  some  flexibility  with  respect  to  equipment 
utilization.  It  has  considerable  flexibility  with  respect  to  the  coordina¬ 
tion  procedures  and  recovery  implementation  methods  that  can  be  used 
by  individual  crews,  and  with  respect  to  the  types  of  problems  simulated. 
Equipment  flexibility  is  provided  by  the  presence  in  the  trainer  of  both 
of  two  distinct  types  of  PAR  found  on  ships  presently  in  operation 
(specifically,  one  Final  Controller  can  use  either  a  SPN  35  or  a  SPN 
42  radar)  and  by  the  built-in  capability  of  using  fewer  than  the  usual 
set  of  three  controller  voice  channels  (a  Marshaller  frequency  and 
two  primary  approach  frequencies,  one  for  each  subteam).  Thus, 
although  the  mockup  does  not  exactly  replicate  any  single  CATCC, 
simulation  realism  is  enhanced  not  only  by  the  use  of  actual  operating 
equipments  to  implement  and  to  display  the  results  of  the  dynamic 
interaction  between  controllers  and  pilots,  but  also  by  the  capabilities 
of  the  trainer  to  somewhat  individualize  training  as  per  the  special 
needs  and  requirements  of  particular  crews  from  particular  ships. 

RECOVERY  PROBLEM  VARIABLES  Mention  has  been  made  above 
of  the  use  of  recovery  problem  variables,  i.  e.  ,  contingency  and 
emergency  situations,  to  vary  recovery  problems  in  the  mockup.  The 
insertion  of  these  problem  variables  by  instructor  personnel  can  serve 
three  functions:  (1)  Training  individuals  and  teams  how  to  deal  with 
each  of  the  different  types  of  problems,  (2)  Training  individual  con¬ 
trollers  to  properly  tradeoff  the  control  requirements  per  aircraft 
within  the  existing  situational  constraints  so  as  to  optimize  the  criteria 
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of  safety  and  time,  and  (3)  Developing  the  higher  levels  of  proficiency 
needed  for  more  difficult  recoveries.  Dae  to  the  impact  of  these 
variables  on  the  analyses  of  collected  performance  data,  they  will  be 
discussed  in  some  detail  here. 

Each  of  the  problem  variables  utilized  in  the  training  device  have 
two  characteristics  in  common:  (1)  They  represent  situations  or 
variables  which  do  occur  during  actual  recovery  operations  and  (2) 

They  alter,  in  one  or  more  ways,  the  performance  requirements 
for  one  or  more  team  members.  Depending  on  which  problem 
variables  are  utilized  during  a  recovery  simulation,  where  in  the 
recovery  profile  they  occur,  subteam  A  vs,  involvement,  and  the 
number,  combination  and  timing  of  these  variables,  various  team 
members  or  combinations  of  team  members  will  be  affected,  or 
stressed,  to  varying  extents.  Examples  served  trainer  recoveries 

are  described  in  Table  2,  providing  illustrations  of  problem  variable 
use  and  how  various  control  positions  may  be  primarily  affected.  Each 
of  the  problem  variables  are  discussed  in  the  paragraphs  below. 

NTDS  SYMBOLOGY  -  This  CATCC  Trainer  has  two  training  respon¬ 
sibilities:  (1)  Training  CATCC  teams  to  effectively  control  the 
recovery  of  aircraft  flights  and  (2)  Introducing  CATCC  crews  to 
NTDS  symbology  and  training  them  how  to  use  it.  The  first  training 
responsibility  was  the  primary  concern  of  this  investigation  and  the 
study  was  accordingly  structured.  However,  due  to  the  facts  that 
(1)  NTDS  symbol  positions  were  manually  controlled  in  the  training 
device  (automatic  tracking  software  programs  are  being  developed) 
thus  constituting  a  new  and  secondary  controller  task,  and  (2)  NTDS 
symbology  is  a  new  development  which  may  or  may  not  benefit  CATCC, 
performance  data  was  identified  by  this  problem  variable  and  con¬ 
trollers  were  questioned  concerning  it. 

SHIP  HEADING  CHANGES  The  approach  path  set  up  for  a  flight 
recovery  is  determined  by  the  expected  path  of  the  ship  during  the 
scheduled  recovery  period.  If  the  ship's  heading  is  changed  with 
little  or  no  advance  notice  to  CATCC,  this  has  two  effects:  (1)  Radar 
displays  require  immediate  corrective  action  and  (2)  Aircraft  require 
timely  information  update  and/or  corrective  control  communications. 
Depending  on  the  point  in  time  during  the  recovery  operation  this 
problem  variable  is  inserted,  it  may  have  minimal  or  extensive  impact 
on  the  controllers. 
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TABLE  2.  EXAMPLES  OF  TRAINER 
RECOVERIES  OBSERVED 


NTDS  Heading  Change  Positions  Number  Total 

Symbology  Time  With  Contingencies,  Primarily  of  No.  of 

Used  Respect  to  Ramp  Emergencies  Affected  Bolters,  Approaches 

Time,  Minutes  W/Os  Completed 


16  before 
24  after 


8  before 


No  changes 


A/C  with 
TACAN  down 


A/C  with  fire 
in  cockpit  ana 
rough  engine 

TACAN  down 


A/C  with 
Radio  down 


A/C  with 
Radio  down 

Ship  officers 
present 


Fouled  deck 


♦  LD  A  ■  Letdown,  or  Approach ,  A 


Marshal, 
LD/A  * 


Marshall, 
LD  B, 
Final  B 

Marshall 
LD  B 


LD  A 


LD  A 


All 

positions 


LD  A,  B 
Final  A ,  B 
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CONTINGENCIES  AND  EMERGENCIES  Contingencies  and  emer¬ 
gencies  can  be  defined  as  unscheduled,  and  hence  unexpected, 
events  which  do  have  some  probability  of  occurrence  associated 
with  them.  Further,  they  are  events  which  require  increased 
attention  and,  often,  additional  and  special  control  actions  on  the 
part  of  team  members.  The  majority  of  the  incidences  included 
within  this  category  relate  to  aircraft  difficulties.  The  few 
exceptions  are  situations  that  were  not  simulated  frequently 
enough  to  constitute  a  major  problem  variable  category  by  them¬ 
selves  (cf.  ,  ship  heading  cha.nges)  but  which  did  impact  on  con¬ 
troller  performance  as  described  above  (e.  g.  ,  fouled  deck).  The 
distinction  between  aircraft  contingency  vs.  emergency  problems 
arises  from  the  need  to  do  whatever  is  feasible  to  recover  an 
emergency  aircraft  immediately.  Examples  of  emergencies 
include  fire  in  the  cockpit  and  rough  running  engine.  Examples 
of  contingencies  include  failed  or  faulty  navigation  and  communica¬ 
tion  equipments.  Contingencies  and  emergencies  primarily  affect 
those  CATCC  positions  in  control  of  the  aircraft;  depending,  how¬ 
ever,  on  the  pattern  of  problem  insertion  where  more  than  one 
event  is  simulated  and  whe'.  her  or  not  the  aircraft  is  recovered  on 
the  first  approach,  the  events  may  secondarily  impact  on  the  rest 
of  the  team  members. 

NUMBER  OF  BOLTERS.  WAYEOFFS.  The  number  of  bolters  and 
waveoffs  occurring  during  the  recovery  of  a  standard  sized  flight, 
when  used  in  conjunction  with  other  information  ruch  delta  instruc¬ 
tions  given,  provides  an  indication  of  air  traffic  density  and  pattern 
complexity  and  of  the  extent  to  which  the  integration  function  had  to 
be  implemented  (see  Figure  3).  The  trainer  is  programmed  to  trap 
aircraft  which  are  within  certain  position  limits  at  one  mile  from  the 
ramp  and  to  waveoff  those  aircraft  not  within  limits.  The  programs 
further  include  a  random  bolter  function  such  that  at  least  two  to 
five  aircraft  will  enter  ihe  bolter-waveoff  pattern  even  if  all  approaches 
are  controlled  within  the  acceptable  one  mile  limits.  Instructor  per¬ 
sonnel  can  and  do  override  the  programs,  however,  to  reduce  or 
increase  the  load  on  particular  controllers  or  on  the  team  as  a  whole. 
The  use  of  this  problem  variable  is  most  directly  related  to  instructor 
perception  of  student  and  team  levels  of  proficiency  and  cf  what  task 
load  will  best  serve  training  objectives  at  that  point  in  time.  For 
example,  if  a  student  Approach  Controller  has  not  previously  per¬ 
formed  in  this  position  -  and  aircraft  controlled  by  his  subteam  are 
not  within  limits  at  one  mile  -  then  instructor  personnel  may  override 
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the  program  and  trap  these  aircraft  so  as  not  to  overload  this 
student.  On  the  other  hand,  if  a  subteam  is  implementing  the 
control  function  sufficiently  well  so  that  few  aircraft  enter  the 
bolter/waveoff  pattern,  instructor  personnel  may  override  the 
program  by  boltering  aircraft  that  would  have  been  trapped;  thus 
placing  increased  performance  requirements  on  this  subteam  and, 
depending  on  the  situation,  the  other  subteam  and/or  the  team  as  a 
whole. 

The  final  column  in  Table  2,  Total  Number  of  Approaches 
Completed,  describes  observed  recoveries  but  was  not  a  problem 
variable.  Its  meaning  will  be  discussed  below. 

DEVICE  UTILIZATION  Although  the  CATCC  trainer  is  described 
as  a  team  trainer,  it  actually  is  used  in  several  ways: 

a.  As  a  team  training  device,  where  team  members 
are  in  essentially  the  positions  they  are  most 
likely  to  be  assigned  to  during  carrier  recovery 
operations. 

b.  As  a  cross-training  device.  For  example,  con¬ 
trollers  whose  previous  training  and  experience 
have  been  as  Final  controllers  will  perform  in 
an  Approach  or  Marshal  position  during  some 
trainer  recoveries. 

c.  As  a  selection  tool.  The  CCAO  and,  especially, 
the  Supervisor  will  try  the  crew  members  out  in 
several  team  configurations  to  determine  the 
optimal  ones.  This  evaluation  may  be  made  not 
only  for  position  assignments  but  also  for  sub¬ 
team  compositions  (i.  e.  ,  which  Approach  con¬ 
troller  should  be  teamed  with  which  Final  con¬ 
troller?  ). 

d.  For  individual  training,  it  is  assumed  that  more 
practice  in  any  one  position  will  result  in  greater 
position  proficiency,  no  matter  how  the  rest  of 
the  team  membership  varies. 


Its  use  is  further  expanded  by  the  type  of  crew  using  the  device 
These  include  (1)  Newly  formed  crews  which  are  relatively  new  to 
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each  other  (although  2  or  3  members  may  have  served  an  earlier 
cruise  together),  to  carrier  air  traffic  control,  and,  in  many  cases, 
to  the  position  they  expect  to  fill;  and  (2)  Experienced  crews  on 
layover,  where  most  members  worked  together  on  the  previous 
cruise  and  expect  to  fill  the  same  positions  on  the  next  cruise. 

The  latter  type  of  crew  use  the  trainer  more  for  refresher  train¬ 
ing  and  to  develop  procedures  and  proficiency  in  problem  areas 
experienced  during  the  previous  cruise.  Teams  of  this  type  were 
not  observed  during  the  study  time  period,  but  it  can  be  noted  that 
they  apparently  tend  to  define  their  own  training  program  to  a 
greater  extent  than  do  the  newly  formed  crews. 

Device  utilization  in  its  intended  manner,  i.  e. ,  simulation  of 
scheduled  Mode  III  recoveries  from  Marshal  check-in  to  i  icovery 
of  the  final  aircraft,  is  interrupted  or  degraded  by  compute*  and 
software  problems  with  relative  frequency.  Thirty  five  percent  of 
the  trainer  recoveries  had  to  be  terminated  prior  to  completion 
while  several  others  were  completed  in  spite  of  software  problems 
(the  total  percentage  of  recoveries  affected  by  software  problems 
is  estimated  to  be  nearly  sixty  percent).  The  CATCC  trainer  is 
one  of  several  tied  into  a  central  computer  system.  This  system 
is  modularized  to  some  extent  with  respect  to  each  of  the  separate 
trainers.  As  a  result,  recovery  problems  may  be  terminated  prior 
to  completion  whenever  software  or  hardware  problems  develop 
within  either  the  CATCC  subsystem  or  the  overall  computer  system. 
Thus,  although  some  amount  of  training  is  provided  on  these  problems 
(amount  of  training  depending  on  amount  of  problem  completed),  it  is 
something  less  than  what  could  be  provided  if  u.ase  failures  did  not 
occur. 


STUDY  METHODS 

STUDY  DESIGN  The  possible  situations  and  schedules  for  data  col¬ 
lection,  i.  e. ,  the  possible  study  design,  were  determined  by  the  type 
of  training  program,  the  criticality  and  nature  of  the  tasks  being 
trained  for,  training  and  operational  schedules,  and  available  student 
populations  (see  pp.  2-4  and  17,  and  Table  1  for  pertinent  dis¬ 
cussions).  The  study's  data  collection  periods  are  indicated  in  the 
following  r  ?trix: 
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TRAINER  RECOVERIES.  Week  of  Training  j  SHIP  RECOVERIES 


Fifth 


SHIP  ONE 


SHIP  TWO 


First 

Third 

X 

X 

This  can  be  roughly  translated  into  experimental  design  terms  as  a 
2x2  factorial  with  independent  measures  of  the  first  factor  and 
related  measures  on  the  second  factor.  The  first  factor  is  amount 
of  training  received  at  two  levels:  one  week  of  training  vs.  approxi¬ 
mately  five  weeks  of  training  (data  representative  of  a  second  week 
of  training  is  available  on  a  very  few  measures  and  is  a  deviation 
from  the  2x2.  See  p.  17  for  a  more  detailed  discussion  of  amount  of 
training  actually  received.  )  The  second  factor  is  performance 
environment  with  trainer  and  ship  being  the  two  environments. 
Repeated  within  cell  measures  of  the  same  performance  indices 
were  obtained  allowing  both  rate  and  extent  of  learning  within  the 
training  environment  to  be  evaluated. 

As  is  often  the  case  in  training  device  evaluations,  a  control 
group  (i.  e. ,  a  group  performing  ship  recoveries  without  having 
received  previous  training)  could  not  be  measured  to  provide  a 
complete  assessment  of  transfer  of  training  (CATCC  is  entirely 
too  critical  to  the  success  of  operational  recoveries  to  purposely 
incur  the  risk  of  poor  performance).  Steps  were  taken,  however, 
to  allow  an  evaluation  of  adequacy,  or  acceptability,  of  performance 
during  the  fi^st  operational  recoveries  which  were  like  those 
practiced  in  the  trainer.  These  steps  included  not  only  the  col¬ 
lection  of  operational  data  on  the  same  parameters  measured  in 
the  device  but  also  the  collection  of  performance  evaluation  data 
from  personnel  able  to  knowledgeably  evaluate  CATCC  adequacy. 
Underlying  the  decision  to  evaluate  operational  performance 
adequacy  was  the  supposition  that  transfer  of  training  does  or  can 
occur  and  it  represented  an  attempt  to  determine  the  adequacy  of 
that  transfer. 

As  discussed  above,  transfer  of  training  cannot  be  conclusively 
demonstrated  in  this  study.  It  will,  however,  be  considered  a 
reasonable  inference  tc  the  extent  that  the  following  results  occur: 

(1)  Differences  in  operational  performance  levels  between 
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teams,  subteams  and  individuals  are  as  would  be  predicted  based 
on  differences  in  amount  of  training  received.  It  is  understood 
that  these  differences  may  be  confounded  by  individual  and  other 
differences  to  the  extent  that  control  groups  do  not  exist  and 
sample  Ns  are  small.  (2)  Unplanned  situations  arise  which  do 
provide  something  akin  to  a  control  group.  These  occurrences 
will  be  pointed  out  in  the  Results  Section. 

Additional  factors,  or  conditions,  which  might  affect  perform¬ 
ance  levels  and  thus  confound  the  results  of  the  study  were  either 
controlled,  randomized,  or  measured  and  incorporated  into  the 
data  analyses.  These  are  discussed  on  pp.  39  -40. 

STUDY  MEASURES  Measurement  data  collected  during  the 
investigation  consisted  of  three  types:  (1)  Objective  and  sub¬ 
jective  controller  performance  data,  (2)  Evaluative  data  on  both 
controller  performance  and  the  training  device,  and  (3)  Descriptive 
data  related  to  recovery  conditions  and  events.  Rationale  and 
definitions  for  each  of  these  sets  of  measures  follows. 

OBJECTIVE  PERFORMANCE  MEASURES  CATCC  constitutes 
an  essential  part  of  the  overall  CATC  system,  a  part  which 
becomes  especially  important  during  Mode  III  recoveries  when 
its  personnel  are  charged  with  direct  implementation  of  system 
objectives.  Under  these  conditions,  it  is  reasonable  to  hypothesize 
a  rather  direct  relationship  between  CATCC  personnel  performance 
levels  and  satisfaction  of  system  objectives  which  are  described  by 
effectiveness  criteria.  This  is  relevant  to  the  CATCC  trainer 
evaluation  in  that  overall  training  objectives  are  consonant  with 
certain  CATC  system  effectiveness  criteria  (see  pp.  I -3  for  dis¬ 
cussions  of  training  objectives  and  the  CATC  system).  If  data  can 
be  collected  on  measures  of  student  performance  which  are  related 
to  these  criteria,  then  any  value  in  device  utilization  demonstrated 
by  improved  performance  levels  will  be  increased  due  to  the  relevance 
of  these  performance  levels,  not  only  to  training  objectives,  but  also 
to  successful  carrier  recovery  operations.  To  do  this,  however, 
requires  that  measures  be  developed  which  describe  the  performance 
of  both  the  controller  and  the  system  or  which,  while  describing  only 
one,  can  be  related  to  the  status  of  the  other. 

The  development  of  such  measures  for  controller  performance 
has  been  a  difficult  problem.  Even  though  it  is  understood  that  air 
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traffic  movement  wilL  be  optimized  to  the  extent  that  the  factors  of  pilot 
behavior,  equipment,  environment  and  controller  performance  are  opti¬ 
mized,  measures  of  controller  performance  which  related  to  system 
effectiveness  criteria  have  been  slow  to  develop.  The  problem  stems, 
at  least  in  part,  from  the  fact  that  implementation  of  the  control  function 
must  be  contingent  to  a  large  extent  on  the  specific  situation  -  and 
situational  conditions  can  vary  tremendously.  In  addition,  the  relation¬ 
ships  between  individual  controllers,  controller  teams,  and  their  equip¬ 
ments  are  generally  complex,  multiple,  interdependent  and  variable; 
further,  the  observable  procedural  implementation  of  control  functions 
tend  to  vary  not  only  as  a  function  of  situational  but  also  individual  pre¬ 
ference  and  capability  As  a  result,  the  tendency  has  been  to  use  either 
quite  global  qualitative  measures  or  very  detailed  quantitative  measures. 
These  provide  data  which  tend  to  be  imprecise  and  unreliable  (in  the  case 
of  qualitative  measurement)  and  without  clear  or  immediate  relationships 
to  system  effectiveness  criteria  (in  either  case). 

Recently,  however,  a  milestone  study  in  the  area  of  controller  per¬ 
formance  developed  a  measurement  set  which  successfully  measured 
the  performance  of  the  individual  controller  on  parameters  which  clearly 
described  ATC  system  effectiveness  levels.  This  effort  was  performed 
by  Buckley,  et  al,  at  NAFEC  (Ref.  2).  Using  a  set  of  simulated  air 
traffic  control  environments  for  one  controller  position,  they  determined 
that  data  collected  on  certain  "system  level"  measures  were  reliably 
determined  by  identity  of  individual  controller  and  system  load  (specifi¬ 
cally,  aircraft  density).  I.  e.  ,  they  found  that  da'a  on  measures  such  as 
number  of  aircraft  conflictions ,  aircraft  time  in  the  system,  and  flight 
time  deviation  for  completed  flights  varied  as  a  function  of  the  parameters 
of  load  condition  and  the  identify  of  the  controller.  Further,  the  authors 
commented  that  ". .  .  there  are  wide  differences  in  the  skill  with  which 
controllers  perform  when  handling  identical  traffic  under  identical  system 
conditions  in  air  traffic  control  simulation  exercises."  In  short,  the 
Buckley  et  al.  effort  provided  an  indication  that  certain  system  level 
measures  were  sensitive  to  variations  in  controller  performance  level 
and  might  vary  as  a  function  of  controller  skill  level. 

For  the  purposes  of  this  study,  therefore,  it  was  hypothesized  that 
CATCC  personnel  performance  is  a  major  factor  determining  CATC 
system  performance  effectiveness  in  both  simulated  and  operational 
environments  and  that  data  collected  on  measures  of  personnel  perfor¬ 
mance  which  defined  or  were  related  to  system  effectiveness  criteria 
would  vary  as  a  function  of  learning.  Criteria  for  successful  carrier 
recovery  operations  which  are  consonant  with  training  objectives  include: 
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a.  Minimum  Recovery  Time  -  necessary  for  reasons  of  tactical 
control  and  effective  use  of  the  carrier  as  a  base  for  air 
operations. 

b.  Minimum  Accident  Rate  -  required  to  maintain  air  operations 
capabilities  . 

c.  Optimized  Carrier  Subsystem  Interfaces  -  CATCC  operations 
interface  with,  and  must  be  coordinated  with,  other  carrier 
operations. 

As  has  been  previously  observed  and  described  (pp.  3  and  29-30) 
the  CATCC  training  device  can  be  evaluated  not  only  as  a  team  trainer 
but  also  as  a  subteam  and  as  an  individual  control  position  trainer. 
Further,  it  was  determined  that  the  relationships  between  CATCC  per¬ 
sonnel  and  CATC  system  objectives  occur  and  are  measurable  at  the 
team,  subteam,  and  individual  position  levels.  It  was  therefore  appro¬ 
priate  to  specify  the  above  learning  hypothesis  to  these  three  personnel 
levels. 

The  objective  performance  measures  are  listed  in  Tables  3  and  4 
and  detailed  definitions  are  provided  both  in  Results,  Section  III,  and 
Appendix  3.  The  tables  serve  to  relate  each  of  these  measures  to  appro¬ 
priate  system  effectiveness  criteria  and  to  appropriate  personnel  level. 

It  will  be  noted  in  review  of  the  tables  that  some  measures  appear  under 
the  special  heading  of  "task"  rather  than  "system".  These  measures  of 
controller  task  performance  impact  on  system  operations,  but  do  not 
necessarily  determine  criterial  levels;  that  is,  poor  or  omitted  perfor¬ 
mance  will  not  necessarily  cause  system  degradation  and  good  controller 
performance  will  not  insure  effective  system  performance.  It  will  also 
be  noted  that  Table  3  lists  two  measures  which  were  subjectively,  rather 
than  objectively,  scored  (Communications  Efficiency,  Communications 
Noise  Level).  These  two  measures  have  been  retained  within  this  group 
rather  than  being  placed  under  Evaluative  Performance  Measures  because 
they  (1)  were  a  part  of  the  measurement  set  collected  on  every  recovery 
(whereas  "evaluative"  measures  were  not)  and  (2)  described  more  dis¬ 
crete  aspects  of  controller  performance  (as  opposed  to  the  more  global 
"evaluative"  measures). 
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EVALUATIVE  PERFORMANCE  MEASURES  In  addition  to  objective 
performance  measurement  as  described  above,  performance  evaluative 
ratings  and  questionnaire  and  interview  responses  were  obtained  from 
student  controllers,  pilots,  Landing  Signal  Officers  (LSOs),  and  the 
Operational  Readiness  Inspection  (ORI)  team.  These  data  served  basically 
two  purposes:  (1)  Provision  of  evaluations  of  the  adequacy  of  controller 
performance  during  operational  recoveries  and  (2)  Explanations  of  events, 
possible  causes,  and  possible  remedies;  both  of  these  as  perceived  by 
intimately  involved  and  cognizant  personnel. 

The  evaluations  by  student  controllers  and  by  pilots  were  made  by 
use  of  three  and  five  point  rating  scales  (ranging  from  Poor  to  Excellent) 
to  describe  controller  performance  level  with  respect  to  Communications, 
Guidance,  Emergency  Procedures,  Contingency  Handling,  and  Overall 
Performance.  The  pilots  were  required  to  do  this  for  each  position: 

Marshal,  Letdown,  and  Final.  In  addition  to  performance  ratings,  eval¬ 
uation  packages  for  both  the  student  controllers  (Controller  Evaluation 
Form)  and  the  pilots  (Pilot's  Evaluation  of  Exercise)  contained  open- 
ended  questions  regarding  possible  problem  areas,  suggestions,  and 
comments.  These  forms  arc  presented  in  Appendix  4. 

The  evaluations  by  LSOs  and  the  ORI  team  are  derived  from  inter¬ 
view  responses.  A  form  was  developed  for  the  LSOs  (see  Appendix  4, 

LSO  Evaluation  of  Exercise)  which  provided  useful  information,  but, 
unfortunately,  too  few  of  the  forms  were  completed  to  provide  sufficiei  ‘ 
data  for  discussion  in  the  Results  Section. 

EVALUATIVE  TRAINING  DEVICE  MEASURES  Both  instructors  and 
students  completed  CATCC  Trainer  Evaluation  forms  (see  Appendix  4) 
which  provided  ratings  of  trainer  realism  and  effectiveness  character¬ 
istics,  verbal  responses  explaining  the  basis  for  the  ratings,  and  sugges¬ 
tions  for  improvement.  All  questions  requiring  a  rating  response  pre¬ 
sented  a  five  point  continuous  scale,  with  end  points  designated  as  Low 
and  High  Effectiveness  or  Realism.  Instructor  evaluations  were  made 
for  team  training  as  a  whole,  while  students  evaluated  the  trainer  in 
terms  of  their  probable  position  at  sea. 

Evaluations  of  training  device  effectiveness  were  required  for  the 
following:  overall  effectiveness  as  a  training  device,  development  of  crew 
coordination  and  skills,  recovery  problems  and  problem  characteristics 
for  training  purposes,  and  compared  to  onboard  training.  In  addition, 
space  was  provided  to  list  effective  and  ineffective  characteristics  which 
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had  not  been  identified  by  responses  to  other  questions.  Questions  were 
also  asked  regarding  the  effectiveness  and  usage  of  NTDS  symbology. 

Evaluations  of  training  device  realism  were  required  for:  target 
and  radar  symbology  presentation  on  the  displays,  target  behavior  on  the 
displays,  situational  response  to  controller  performance,  recovery  pro¬ 
blems,  communication,  guidance  control,  contingency  handling,  emergency 
situations,  and  the  controller's  own  performance.  A  final  question  was 
asked  regarding  any  additional  comments,  suggestions  or  statements  the 
respondee  might  wish  to  make. 

DESCRIPTIVE  DATA  If  the  conditions  under  which  operators  perform 
and  the  nature  of  the  problem  they  deal  with  are  constant,  then  operator 
performance  levels  should  be  determined  only  by  operator  characteristics  - 
such  as  amounts  of  training  received.  To  the  extent  conditions  and  the 
problem  are  variable,  however,  these  variations  may  tend  to  affect  per¬ 
formance  levels.  A  number  of  condition  and  problem  variables  may 
affect  CATC  system  performance  levels  depending  on  whether  or  not 
they  occur  and  the  extent  to  which  thev  occur;  descriptive  data  were  col¬ 
lected  on  these.  It  was  expected  that  certain  of  these  would  require 
consideration  during  data  analysis.  It  was  hoped  that  the  others  would 
either  not  vary  to  any  extent  (i.  e.  ,  be  controlled  variables)  or  would  vary 
equivalently  under  the  conditions  to  be  compared  (i.  e.  ,  be  randomized 
variables).  As  it  turned  out,  variables  behaved  as  expected  and  hoped. 
They  can  be  separated  into  variable  groups  which  serve  to  identify  them: 

a.  Problem  variables  (see  pp.  26-30). 

b.  Other  significant  events  -  for  example,  unexpected  equipment 
failures,  failures  to  respond  or  incorrect  responses  on  the  part 
of  pilots  to  correctly  given  control  instructions. 

c.  Experience  levels  for  the  controller,  pilot  (both  PC&E  and 
carrier),  and  deck  recovery  crew  personnel. 

d.  Weather  conditions. 

Time -based  records  were  kept  for  variables  falling  into  groups  a  and  b 
throughout  each  recovery  and  were  later  used  in  conjunction  with  data 
analysis.  It  was  determined  from  data  collected  on  group  c  and  d  vari¬ 
ables  that  these  variables  had  been  either  controlled  or  randomized. 

For  example,  PC  8t  E  pilot  experience  level  was  controlled  throughout 
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the  study.  On  the  other  hand,  it  was  determined  that  carrier  pilot 
experience  level  was  randomized  in  that  the  pilot  population  for  each 
carrier  recovery  constituted  an  independent  and  equivalent  sample  of 
carrier  pilot  experience. 

DATA  COLLECTION  METHODS  The  data  reported  in  this  study  were 
collected  by  two  Bunker  Ramo  human  factors  personnel  teamed  with  one 
to  two  FAAWTC  instructor  personnel,  depending  on  availability.  Which 
personnel  collected  data  on  which  measures  was  dependent  on  the  level 
of  cognizance  required  and  available  sources  of  information  at  each  data 
collection  position.  Certain  measures  required  little  experience  (e.  g.  , 

Marshal  Information  Broadcast:  Accuracy  and  Completeness)  while 
others  required  a  substantial  amount  (e.  g.  ,  ratings  of  Communications 
Efficiency).  Instructor  coverage  on  the  latter  measures  was  provided 
throughout  the  study  to  the  extent  possible,  with  the  most  complete 
coverage  being  provided  during  the  early  data  collection  periods. 

The  descriptive  and  quantitative  information  providing  the  measure¬ 
ment  data  was  obtained  by  monitoring  the  control  communication  channels, 
extra-channel  communications,  radar  screens,  Marshal  and  Final  status 
boards,  and  the  mission  clock  or  other  correlated  time  piece.  ) 

DATA  ANALYSIS  METHODS  Graphic  analysis  was  selected  as  the  most 
appropriate  and  efficient  method  for  identification  of  variables  contributing 
to  data  variations  and  subsequent  development  of  learning  curves.  This 
decision  was  based  on  the  facts  that  (1)  the  CATCC  training  program 
emphasized  continued  training  of  the  same  activities  to  develop  proficiency, 

(2)  each  measure  was  an  individual  case  in  terms  of  analysis  (due  to  pro¬ 
blem  variable  usage  and  whether  the  measure  related  to  team,  subteam  or 
individual  performance),  and  (3)  each  measure  was  represented  by  small 
data  samples  (due  to  problem  stoppages,  cross  training,  and  inability  to 
always  cover  all  data  collection  positions). 

The  primary  contributors  to  performance  variance  were  expected 
to  be  personnel  differences,  problem  variables,  and,  of  course,  repeated 
use  of  the  training  device.  These  were  indeed  the  major  contributors 
(accounting  for  most  of  the  variance)  and  analysis  methods  related  to  each 
are  discussed  in  the  following  paragraphs. 

The  effect  of  personnel  differences,  i.  e.  ,  the  effect  of  team,  sub¬ 
team  and  individual  identities  on  performance  level,  was  controlled.  That 
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is,  a  constant  personnel  membership  was  maintained  for  each  data  point 
within  a  performance  curve. 

It  was  determined  during  initial  analysis  activities  that  descriptive 
data  on  problem  variables  could  be  transformed  into  levels  of  problem 
difficulty  (c.  g.  ,  "easy11,  "difficult”)  and  that  these  levels  of  difficulty 
appeared  to  account  for,  or  correlate  with,  performance  variations  not 
accounted  for  by  personnel  and  learning  stage  differences.  Once  this 
had  been  determined,  a  level  of  difficulty  was  judged  for  each  measure 
on  each  recovery  problem.  These  judgments  were  made  independently 
of  other  analyses  to  ensure  nonbiased  decisions. 

Separate  difficulty  analyses  per  measure  per  recovery  were  neces¬ 
sitated  by  the  facts  that  (1)  Measures  differed  in  terms  of  activities  and 
positions  measured  and  (2)  The  impact  of  problem  variables  varied  as  a 
function  of  activity  measured  and  the  time  and  conditions  under  which 
they  occurred.  I.  e.  ,  a  recovery  problem  described  as  "difficult"  for  one 
measure  may  be  described  as  "easy"  for  another  measure.  An  illustra¬ 
tive  example  is  provided  by  considering  the  varying  impacts  on  perfor¬ 
mance  that  are  possible  for  one  problem  variable,  Ship  Heading  Change. 

An  unexpected  ship  heading  change  occurring  just  prior  to  ramp  time  would 
not  affect  measures  of  the  Marshal  Information  Broadcast,  as  this  activity 
would  already  have  been  completed.  The  heading  change  would  require 
the  Marshaller  to  make  an  additional  general  information  broadcast  but 
this  did  not  appear  to  impose  any  special  load  on  the  Marshaller.  It 
would  not  be  expected,  therefore,  that  the  aspect  of  Marshaller  perfor¬ 
mance  measured  by  EAT  Deviations  Beyond  t.  25  Minutes  Not  Acted  Upon 
would  be  degraded  either.  On  the  other  hand,  a  heading  change  just  prior 
to  ramp  time  can  impact  on  the  Approach  Controller  measure,  Deviation 
from  Scheduled  Ramp  Time  for  First  Aircraft.  Further,  any  heading 
changes  occurring  throughout  the  interval  between  four  minutes  prior  to 
ramp  time  and  recovery  completion  may  affect  other  measured  perfor¬ 
mances  for  a  period  of  from  two  to  five  minutes  per  heading  change  (e.  g.  , 
Average  Aircraft  Separation  Error).  This  affect  results  from  requirements 
that  may  arise  for  additional  control  decisions,  issuance  of  more  control 
instructions,  and  extra  coordination  efforts  between  team  members.  The 
extent  of  this  effect  on  performance  is  judged  by  consideration  of  the 
occurrence  and  timing  of  other  events  in  the  problem.  For  example,  a 
recovery  may  be  judged  as  "difficult"  for  Subteam  A  because  an  unex¬ 
pected  heading  change  occurred  when  three  of  their  aircraft  were  on  the 
approach  path  within  ten  miles  of  the  carrier,  one  of  which  was  an 
emergency  aircraft.  The  same  recovery,  however,  may  be  judged  as 


41 


NAVTRADEVCEN  70-C-0258-1 


"easy11  for  Subteam  B  because  they  had  fewer  aircraft  under  control, 
none  of  which  presented  pressing  position  or  emergency  control  require¬ 
ments. 


The  effect  of  problem  variables  was  handled  in  one  of  two  ways, 
depending  on  the  comparative  distributions  of  difficulty  levels  across 
data  groupings  developed  for  the  construction  of  a  learning  curve.  If 
the  difficulty  levels  within  a  data  group  (a  point  on  the  learning  curve) 
were  quite  similar,  but  varied  across  the  data  groups  for  that  curve, 
and  that  variation  correlated  with  performance  level  variations,  then  the 
difficulty  level  was  graphically  depicted.  If,  instead,  difficulty  levels 
varied  randomly  within  data  groups  and  the  distributions  were  similar 
across  the  groups  then  any  problem  variable  effects  were  considered  to 
be  randomized  with  similar  average  levels. 

The  majority  of  the  remaining  performance  variance  was  accounted 
for  by  the  variable  of  primary  concern,  amount  of  training  device  usage, 
or  number  of  recoveries  performed.  This  variable  (and  number  of  opera¬ 
tional  recoveries  performed)  provider,  the  independent  variable  axis  for 
graphs  presented  in  the  Results  section. 

Mention  should  be  made  of  data  reduction  procedures  used  to  ensure 
data  which  reflected  controller  performance  rather  than  the  operation  of 
other  factors.  In  some  instances,  in  both  trainer  and  operational  envi¬ 
ronments,  pilots  either  did  not  respond  or  responded  incorrectly  to  con¬ 
troller  instructions.  The  data  for  these  instances  was  deleted  from  the 
data  base  under  analysis.  For  example,  if  one  of  the  five  or  six  aircraft 
under  the  control  of  Subteam  A  did  not  respond  to  correctly  given  glideslope 
and  azimuth  corrections  then  the  measured  error  at  one  mile  for  this  air¬ 
craft  was  not  considered  in  the  derivation  of  mean  position  errors  for  that 
subteam. 
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APPENDIX  B 

DETAILED  STUDY  RESULTS 

INTRODUCTION 

The  empirical  results  of  the  investigation  are  presented  and  discussed 
in  the  following  paragraphs  and  associated  figures.  Results  derived  from 
the  objective  performance  measurement  data  are  depicted  graphically  with 
the  independent  axis  identifying  the  number  of  recoveries  performed  in  the 
trainer  per  ship  crew  and  in  the  operational  environment.  It  will  be  re¬ 
called  (see  p.  17  )  that  utilizablc  data  for  Ship  One,  v/ith  few  exceptions, 
describes  recoveries  performed  during  the  first  week  of  training.  The 
exceptions  consist  of  three  measures  with  data  which  could  be  described 
as  second  week  performances.  The  data  presented  for  Ship  Two  will 
reflect  performances  during  something  less  than  the  fifth  week  of  training. 
The  results  derived  from  the  evaluative  measurement  data  will  be  pre¬ 
sented  in  tabular  and  bar  graph  form. 

OBJECTIVE  PERFORMANCE  MEASURES 

The  learning  curves  will  be  presented  in  three  groups,  representing 
team,  subteam  and  individual  performance.  It  will  be  observed  that  the 
graphs  generally  display  improved  performance  as  a  function  of  repeated 
performance  within  each  set  of  recoveries  (i.  e.  ,  first  week  for  Ship  One, 
fifth  week  for  Ship  Two,  operational  recoveries  for  both  ships).  It  will 
also  be  observed  that  performance  levels  generally  improve  across  the 
two  sets  of  trainer  recoveries  and  that  Ship  Two  performed  better  in 
most  respects  during  operational  recoveries  despite  difficult  recovery 
problems.  As  the  crews  from  only  two  ships  were  observed  at  different 
points  in  training  these  observations  do  not  constitute  conclusive  evidence 
that  five  weeks  of  training  results  in  better  performance  than  one  week; 
however,  observation  of  the  teams  in  the  trainer  and  ship  environments 
supports  the  conclusion  that  the  measured  performance  differences  were 
primarily  the  result  of  training  differences.  Relevant  to  the  comparison 
of  data  for  predictive  purposes  is  the  observation  by  instructor  personnel 
that  the  crew  from  Ship  Two  had  performed  better  during  their  fourth  week 
of  training  than  they  did  during  the  observed  fifth  week. 
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TEAM*  MEASURE:  AVERAGE  AIRCRAFT  SEPARATION  ERROR  Thfe 
data  appearing  in  Figures  5  and  6  describe  the  average  absolute  devia¬ 
tion  from  an  acceptable  time  interval  between  aircraft  (1.  0  ±  .  5  or  2.  0 
t  .  5  minutes)  as  measured  at  the  carrier  ramp.  **  As  can  be  seen  by 
I  examination  of  Figure  5.,  two  parameters  were  found  to  determine  the 

|  mean  performance  levels  for  this  measure:  (1)  Skill  level,  as  described 

i  by  number  of  recoveries  performed  as  a  team,  and  (2)  Recovery  difiV  - 

l  culty  level.  As  can  be  seen  by  examination  of  Figure  6  ,  which  presents 

an  individual  curve  for  each  team,  team  identity  was  a  third  important 
determinant  of  aircraft  separation  error.  Thus,  the  hypothesis  of  team 
learning  in  the  device  appears  to  be  supported  and  the  effect  of  two  other 
variables  on  performance  level  has  been  described. 

| 

j  The  question  of  whether  or  not  transfer  of  training  occurred  can  best 

!|  be  evaluated  in  terms  of  Figure  5,  where  selected  individual  Ship  One 

3  team  curves  can  be  compared  with  Ship  Two  team  curves  (see  pp.  17-19) 

ij  regarding  the  comparative  compositions  of  teams  observed  in  the  trainer 

i  vs.  onboard).  The  Ship  One  teams  selected  for  comparison  are  Teams 

Six  (T6,  Trainer  Recoveries)  and  Ten  (T 10,  Operational  Ship  Recoveries). 

|  Of  the  Ship  One  teams  providing  trainer  data  for  this  measure  (e.  g.  ,  Tl, 

T7),  Team  Six  was  the  most  similar  to  Team  Ten  in  terms  of  individual 
device  training  received,  proficiency  demonstrated  and  previous  exper¬ 
ience. 

If  positive  transfer  of  training  occurred  and  the  operational  perfor¬ 
mance  level  was  a  function  of  the  amount  of  training  received  then  the 
Ship  One  team  should  have  performed  more  poorly  than  the  Ship  Two  team 
j  in  the  operational  environment  (they  received  less  device  training  and  none 

i  as  a  team)  -  if  the  recovery  problems  were  similarly  difficult.  The  on¬ 

board  rt  overy  problems  were  not,  however,  similarly  difficult.  The 
operation-T  recoveries  performed  by  the  Ship  Two  team  were  judged  con¬ 
siderably  more  difficult  than  those  performed  on  Ship  One  due  to  radar 
malfunctions  that  began  during  the  third  recovery,  worsened  during  the 
|  fourth  recovery,  and  continued  during  the  fifth  and  last  recovery  observed, 

j  The  radar  problem  was  compounded  by  aircraft  and  other  contingency 


1 

f 

5 

i 


*  A  team  is  defined  by  constant  composition  with  respect  to  member¬ 
ship  and  position.  I.  e.  ,  if  the  controllers  in  Final  and  Marshal 
exchange  positions,  a  new  team  has  been  formed. 


♦♦More  detailed  definitions  are  given  for  some  measures  in  Appendix  3. 
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conditions.  Thus,  if  positive  transfer  of  training  did  not  occur,  and 
performance  on  this  measure  is  as  sensitive  to  recovery  difficulty  as  it 
appears  to  be,  then  Ship  One  should  have  performed  better  than  Ship  Two. 

With  the  above  in  mind,  it  can  be  observed  in  Figure  6  that  the  Ship 
One  team  (T6,  T10)  received  less  training  and  performed  poorer  than  the 
Ship  Two  team  in  the  shipboard  environment.  The  amount  of  data  and 
conditions  of  measurement  are  insufficient  to  support  any  firm  conclu¬ 
sions  regarding  transfer  of  training  on  this  measure  (e.g.  ,  team  differ¬ 
ences  may  have  confounded  the  results).  It  can  only  be  said  that  the 
measured  performance  differences  are  definitely  in  the  direction  re¬ 
quired  by  the  transfer  of  training  hypothesis. 

Although  a  criterion  is  established  for  aircraft  separation  (1.  0  t  .  5 
or  2.  0  *  .  5  minutes),  no  operational  data  was  located  which  described 
deviations  from  this  criterion  as  measured  in  this  study.  As  a  result, 
evaluation  of  CATCC  operational  performance  adequacy  on  Ships  One  and 
Two  must  be  based  on  judgment,  rather  than  on  comparison  with  known 
data  (see  p.  32  for  adequacy  evaluation  rationale).  It  would  appear  that 
the  mean  separation  errors  for  the  Ship  Two  team  (.27,  .  35,  and  .  08 
minutes)  were  quite  acceptable,  especially  in  view  of  the  fact  that  the 
errors  resulted  from  an  intentional  increase  in  spacing  between  aircraft 
when  radar  presentations  began  to  degrade.  In  contrast,  separation  does 
r.ot  appear  to  he  been  adequately  controlled  by  the  Ship  One  team.  The 
error  /alue  on  t.  third  recovery  was  .  72  minutes  and  the  amount  of 
separation  between  aircraft  was  quite  inconsistent  (e.  g.  ,  1.  75,  4.  00,  0,  50 
and  2.  25  minutes  describes  spacing  between  five  aircraft  in  sequence  on 
this  recovery).  The  ,  27  error  value  obtained  on  the  fourth  recovery  by 
this  team  appears  to  represent  an  adequate  performance,  particularly 
since  the  spacing  was  more  regular  and  the  measured  error  describes  a 
maintenance  of  increased  spacing  between  aircraft.  In  summary,  the 
team  with  more  training  performed  quite  acceptably;  the  team  with  less 
training  did  not  do  as  well. 

One  final  note  of  interest  (the  same  phenomenon  can  also  be  observed 
on  the  next  team  measure)  is  the  direction  of  the  individual  team  curve 
for  To  for  their  two  trainer  recoveries  (Figure  6).  If  previous  individual 
training  does  transfer  to  the  team  situation  (training  programs  generally 
presume  this  does  happen  to  some  extent  at  least)  then  the  fact  that  T6 
performed  its  first  recovery,  a  difficult  one,  at  a  level  similar  to  the 
easy  first  recoveries  performed  by  Tl,  T2  and  T3  is  not  surprising 
this  recovery  was  performed  on  the  final  day  of  training  for  this  ship's 
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cri'w.  What,  is  of  interest  is  the  sharp  degradation  of  performance  on 
T6's  second  recovery.  This  measure  of  team  performance  does  appear 
to  be  sensitive  to  recovery  difficulty,  but  note  that  the  variation  of  perfor¬ 
mance  level  in  response  to  difficulty  by  the  Ship  Two  team  during  their 
fifth  week  of  training  is  not  nearly  as  extreme.  A  possible  reason  of 
T6's  extreme  performance  degradation  may  be  the  difference  in  controller 
and  team  self-confidence  and  capability  with  respect  to  difficult  problems 
when  one  vs.  five  weeks  of  training  have  been  received.  This  point  will 
be  discussed  again  with  reference  to  evaluative  data  (pp.  85-86). 

It  will  be  noted  that  certain  of  the  data  points  in  Figure  6  describe 
trainer  recoveries  in  which  NTDS  syn.bology  was  utilized  by  the  control¬ 
lers.  If  the  downward  trend  of  the  learning  curve  s  hypothesized  to  con¬ 
tinue  until  performance  is  perfected,  then  it  would  be  expected  that  the 
separation  error  on  recoveries  4  and  5  made  by  Ship  One's  T1  (these  two 
recoveries  were  made  even  easier  by  use  of  a  two  minute  interval  between 
aircraft)  and  on  recovery  5  made  by  Ship  Two's  team  would  have  been 
smaller  than  they  were.  As  noted  earlier,  this  result  does  not  necessarily 
evaluate  the  utility  of  NTDS  symbology  --  but  it  does  suggest  that  imposi¬ 
tion  of  a  secondary  task  on  the  controller  may  affect  performance  of  a 
primary  function. 

TEAM  MEASURE:  TIME  TO  COMPLETE  RECOVERY  OF  11  AIRCRAFT 
This  team  measure  is  defined  ao  the  time  interval,  in  minutes,  from 
Ramp  Time  to  the  trapping  of  the  final  scheduled  aircraft.  Certain  de¬ 
scriptive  data  were  used  in  the  data  reduction  process  for  this  measure; 
specifically,  periods  during  which  recovery  operations  were  suspended 
(e.  g.  ,  due  to  a  fouled  deck)  and  delays  caused  by  the  failure  of  other 
personnel  groups  to  communicate  essential  information  were  subtracted 
from  the  total  recovery  time. 

This  particular  measure  is  one  which  was  not  expected  to  be  sensitive 
to  changes  in  controller  team  skill  level  except,  perhaps,  in  the  grossest 
sense.  Once  the  obvious  data  corrections  described  above  were  made, 
the  time  required  to  complete  a  scheduled  recovery  was  expected  to  be 
primarily  a  function  of  other  variables,  many  of  which  are  reflected  in 
the  bolter /waveoff  rate  (e.  g.  ,  pilot  skill,  weather  conditions,  LSO  de- 
cirion  criteria). 

The  reason  this  measure  was  of  interest,  however,  was  because  it 
appeared  to  tap  an  aspect  of  control  team  performance  not  covered  by  the 
other  measures.  Other  measures  in  the  objective  measurement  set 
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primarily  describe  the  effectiveness  of  aircraft  control  with  respect  to 
integration  into  the  approach  path  {separation  error)  or  control  on  that 
path  (separation  error  and  glidepath  errors)  and  the  effectiveness  of 
communications  behaviors  utilized  to  effect  or  make  control  decisions. 

An  additional  and  very  important  aspect  of  controller  team  behavior  is 
the  decision-making  management  activities  required  to  select  the  course 
of  action  for  a  developing  or  sudden  problem,  to  develop  traffic  patterns 
suitable  to  a  given  situation,  and  to  perform  the  trade-offs  that  must 
usually  be  considered.  It  was  felt  that  the  quantitative  measure  most 
likely  to  be  consistently  affected  by  this  behavioral  aspect  would  be  re¬ 
covery  duration.  For  this  reason,  data  for  this  measure  was  analyzed  -- 
with  surprising  results. 

The  results  of  the  analyses  indicate,  first  of  all,  that  although  bolter/ 
waveoff  rate  may  be  strongly  deterministic  of  recovery  completion  time 
when  controller  skill  level  and  recovery  difficulty  are  held  constant  -- 
it  is  not  when  these  two  parameters  are  allowed  to  vary.  The  data  points 
depicted  in  Figures  7  and  8  represent  recoveries  which  range  in  number 
of  approaches*  from  12  to  26  quite  randomly.  Despite  this,  the  majority 
of  the  variance  for  data  on  this  team  measure  is  accounted  for  by  skill 
level,  recovery  difficulty  level,  team  differences  anda  correction  factor 
(  for  two-minute  separations.  **  Whether  or  not  these  factors  would  account 

for  the  majority  of  variance  under  more  usual  conditions  remains  to  be 
seen,  but  it  is  of  interest  that,  when  allowed  to  vary,  they  appear  to  exert 
such  an  influence. 

The  comments  made  on  the  previous  measure  regarding  team  com¬ 
positions  (T6,  T10)  and  recovery  difficulty  levels  as  related  to  transfer- 
of-training  considerations  are  applicable  here  when  reviewing  Figure  8, 

And  the  same  evaluations  regarding  impact  of  team  training  on  operational 
performance  can  be  made,  with  some  further  support  as  described  in  the 
following  paragraph. 

*  Number  of  Approaches  =  number  of  aircraft  recovered  4-  number  of 
*  deck  bolters /waveoffs  +  number  of  bingos /go -a rounds /waveoffs  initiated 

prior  to  deck. 

** As  can  be  seen  by  reference  to  Figures  7  and  8,  certain  recoveries  utilized 
a  two-  rather  than  a  one-minute  separation  between  aircraft.  A  hypo¬ 
thetical  correction  factor  of  ten  minutes  (a  judgment  only,  based  on 
several  factors)  was  subtracted  to  create  adjusted  curves  which  are 
more  comparable  to  the  recoveries  using  one -minute  separations. 
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An  additional  data  point  was  obtained  on  this  measure  which 
described  performance  by  the  first  team  observed  on  Ship  One  perform¬ 
ing  an  easy  recovery  (T9,  Figure  8  ).  It  will  be  recalled  that  this  team 
had  two  controllers  who  had  not  received  device  training  and,  in  addition, 
jj  had  not  received  device  training  as  a  team.  Even  though  the  recovery 

was  the  easiest  one  observed,  measured  performance  was  rather  poor 
•  (see  following  paragraph  re:  acceptable  recovery  durations).  If  recovery 

|  difficulty  affects  this  team  measure  as  it  appears  to,  then  an  even  poorer 

!  performance  would  have,  occurred  on  a  difficult  recovery.  Again,  the 

|  difference  between  teams  in  operational  performance  levels  is  as  required 

;  by  the  transfer  of  training  hypothesis.  From  the  limited  evidence  avail- 

;  able,  it  appears  that  training  of  all  team  members  as  a  team  prior  to 

I  on-the-job  performance  would  benefit  recovery  operations  efficiency. 

f 

t 

I 

\  A  sample  of  data  describing  regular  night  recovery  operations  on 

|  another  carrier  was  available  to  provide  a  basis  for  evaluating  the  ade¬ 

quacy  of  Ship  One  and  Two  teams  on  this  measure  (Ref.  1).  Based  on  a 
regression  equation  developed  from  data  on  flights  of  2  to  17  aircraft 
the  expected  recovery  duration  for  11  aircraft  is  25  minutes  with  a  pos¬ 
sible  range  of  12  to  37  minutes  (95%  confidence  interval).  The  actual 
data  for  11  aircraft  recoveries  provided  a  mean  time  period  of  24  minutes 
with  a  range  of  19  to  34  minutes.  These  latter  values  can  be  compared  to 
recovery  time  periods  measured  onboard  Ships  One  and  Two: 


SHIP  ONE 

SHIP  TWO 

..  ,  Two -Minute 

Measured 

Correction 

ANOTHER 

CARRIER 

T9:  31 

33 

33 

34 

T 10:  28 

29 

19 

23 

T10:  29 

26 

16 

21 

19 

Be  sed  on  the  above  data,  Ship  One  and  Two  team  performances  on  this 
measure  in  their  onboard  environments  appear  acceptable.  Again,  how¬ 
ever,  T9's  performance  appears  somewhat  marginal  for  an  easy  recovery 
and  it  must  be  pointed  out  that  this  team  did  not  receive  the  same  training. 
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SUBTEAM*  MEASURE:  AIRCRAFT  POSITION  AT  ONE  MILE:  AVERAGE 
AZIMUTH  ERROR  This  measure  is  defined  as  the  average  absolute  hori¬ 
zontal  deviation  of  CATCC  controlled  aircraft  from  the  zero,  or  perfect, 
position  at  one  mile  from  the  carrier  ramp.  The  within  and  across  down¬ 
ward  trends  for  curves  appearing  in  Figure  9  indicate  reduction  in  azimuth 
error  as  a  function  of  amount  of  training.  Population  sizes  in  each  con¬ 
dition  (except  the  second  one)  are  large  enough  to  begin  to  randomize  any 
effects  from  individual  subteam  differences,  thereby  making  extrapolations 
from  the  first  to  the  fifth  week  of  training  more  justifiable.  The  direction 
of  the  considerable  difference  between  operational  performance  levels  for 
Ships  One  and  Two  subteams  suggests  transfer  of  training  on  this  measure 
to  the  operational  environment.  The  only  curve  in  Figure  9  which  appears 
>  irregular  is  the  second  one  (N  =  1),  but  when  compared  to  the  first  curve 
(the  individual  curve  for  this  subteam  during  the  first  week  of  training  is 
practically  identical  to  the  mean  curve),  the  learning  hypothesis  is  again 
supported.  (The  perfect  performance  on  recovery  two  by  that  team  during 
the  second  week  can  only  be  considered  a  random  error  of  good  fortune. ) 

Figure  10  presents  data  on  identical  subteams  from  Ships  One  and 
Two  to  provide  a  comparison  of  trainer  and  ship  performances.  Figure 
10  suggests  the  training  device  provides  data  predictive  of  operational 
subteam  performance  and/or  provides  subteam  training  which  is  trans¬ 
ferred  to  the  operational  setting. 

The  standard  for  acceptable  azimuth  control  of  the  aircraft  at  one 
mile  from  the  ramp  is  positioning  within  i  100  feet  of  the  glidepath  and, 
with  one  exception,  all  the  observed  subteams  performed  within  this 
standard  during  actual  operations.  In  terms  of  evaluating  impact  of  train¬ 
ing  on  this  measure,  it  is  of  interest  to  note  that  the  one  exception  was 
controlled  by  a  Ship  One  subteam  which  had  not  trained  as  a  subteam  and 
in  which  the  Approach  controller  had  not  received  device  training.  I  urther, 
the  best  Ship  One  subteam  performance  during  recovery  operations  was 
by  the  one  subteam  that  had  trained  as  i  subteam  and  in  which  both  subteam 
members  had  received  position  training  in  the  trainer(depicted  in  Figure 
10).  Comparison  of  that  subteam's  operational  performance  in  Figure  10 
with  the  mean  Ship  One  operational  performance  curve  in  Figure  9  again 
provides  a  basis  for  suggesting  that  this  aspect  of  recovery  operations 
effectiveness  benefits  from  device  training. 

*  A  subteam  is  defined  by  constant  membership  with  respect  to  position; 
e.  g.  ,  if  two  controllers  exchange  Approach  and  Final  positions,  a  new 
subteam  has  been  formed. 
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The  parameters  of  recovery  difficulty  appeared  to  influence  per¬ 
formance  levels  on  both  azimuth  and  glideslope  control  but  the  effect 
was  not  entirely  consistent.  This  inconsistency  may  stem  from  the  need 
of  controllers  to  employ  decision  processes  during  task  stress  conditions 
to  determine  the  relative  emphasis  to  be  given  to  each  of  the  control 
functions.  Such  decisions  are  based  to  some  degree  on  both  the  state 
of  situational  variables  (e,g.  ,  air  traffic  situation,  sea  state,  pilot  be¬ 
havior,  etc.  )  and  personal  experience  with  these  variables.  I.  e.  ,  it  is 
specifically  hypothesized  that  the  relative  emphasis  given  to  azimuth  and 
glideslope  control  functions  during  difficult  recoveries  may  vary  as  a 
function  of  such  decision  processes. 

SUBTEAM  MEASURE:  AIRCRAFT  POSITION  AT  ONE  MILE:  AVERAGE 
GLIDESLOPE  ERROR  Defined  as  the  average  absolute  vertical  deviation 
of  CATCC  controlled  aircraft  from  the  zero,  or  perfect,  position  at  one 
mile  from  the  carrier  ramp.  Again,  the  performance  improvement  trends 
within  the  training  device  environment  are  quite  evident  and  the  population 
sizes  adequate  to  suggest  that  training  beyond  the  first  week  will  result 
in  additional  improvement  in  the  trainer.  The  mean  curves  describing 
operational  recoveries  in  Figure  lido  not,  however,  suggest  that  the 
additional  training  improves  operational  performance  on  this  measure. 
(Which  is  not  to  say  that  no  benefit  is  derived  from  training  on  this  mea¬ 
sure;  evidence  suggesting  the  need  for  some  subteam  training  is  discussed 
in  the  next  paragraph.  ) 

The  operational  standard  for  acceptable  glideslope  control  at  one 
mile  from  the  ramp  is  aircraft  positioning  within  +100  to  -50  feet  of  the 
glidepath.  Ail  of  the  subteams  performed  acceptably  with  respect  to  this 
standard  with  one  exception;  two  approaches  were  below  -50  feet  and  these 
were  controlled  by  the  same  Ship  One  subteam  which  had  also  performed 
unacceptably  on  azimuth  control. 

The  best  Ship  One  performances  (average  error  =  12  feet)  were  by 
two  subteams  both  of  whose  members  had  received  device  training.  One 
pair  trained  together  as  a  subteam  while  the  other  did  not.  The  rest  of 
the  Ship  One  performances  were  much  poorer  (average  error  =  25  feet) 
and  were  by  subteams  containing  a  member  who  had  not  trained  in  the 
device.  From  this  evidunce  it  appears  that  acceptable  operational  per¬ 
formance  is  more  likely  if  at  least  one  week  of  subteam  training  has  been 
received. 
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It  is  of  some  interest  to  review  f i vi r < ; s  9  and  Hand  compare  the 
azimuth  and  glide  slope  error  magnitudes.  First,  it  can  be  observed  that 
glideslope  errors  tend  to  be  smaller  than  azimuth  errors  in  the  trainer 
environment.  The  same  is  true  for  operational  performance:  by  Ship  One 
subteams.  This  might  be  explained  by  the  fac  t  that  glideslope  deviation 
is  more  critical  both  with  respect  to  probability  of  trapping  (due  to  the 
geometry  of  the  situation,  glideslope  deviations  result  in  larger  impact 
point  deviations)  and  accident  probability  (an  airc  raft  much  below  glideslope 
may  strike  the  ramp).  This  greater  criticality  may  result  in  greater 
attention  on  the  part  of  the  controller  and,  possibly,  more  stringent  train¬ 
ing  in  earlier  training  phases. 

Second,  it  can  be  observed  that  the  subteam  glideslope  errors  for 
Ship  Two  operational  reeeoveries  arc*  larger  than  their  azimuth  errors 
but  that  their  mean  performance  levels  are  nut  markedly  different  from 
those  for  Ship  One,  nor  from  what  they  attained  in  the  trainer.  From  this 
and  the  training  discussions  above,  it  follows  that  a  10  to  11  foot  average 
glideslope  error  may  be  the  reasonable  and  attainable  performance  stan¬ 
dard  when  controlling  an  average  pilot  population. 

SUBTEAM  MEASURE:  COMMUNICATIONS  EFFICIENCY  The  two  com¬ 
munications  measures  (Efficiency  and  Noise  Level)  differ  from  the  other 
measures  in  that  they  are  judgments  made  by  instructor  personnel  using 
a  graphic  rating  scale.  Communications  efficiency  is  defined  as  the 
transmittal  of  maximum  information  content  in  a  minimum  amount  of  time 
which  is.  understandable  by  and  necessary  to  the  pilot  under  control.  The 
standard  of  comparison  utilized  by  these  personnel  for  the  top  end  of  the 
scale  was  that  communications  efficiency  level  which  had  been  observed 
for  very  good,  operationally  experienced,  control  teams.  The  only  un¬ 
usual  data  reduction  procedure  utilized  on  this  measure  was  to  separate 
ratings  made  by  instructor  personnel  from  those  made  by  Bunker  Ramo 
personnel.  This  was  considered  advisable  in  that  while  all  of  the  instruc¬ 
tor  personnel  possessed  extensive  operational  and  training  backgrounds 
on  which  to  base  their  criterion  definition  and  use  of  the  scale,  Bunker 
Ramo  personnel  did  not. 

As  can  be  seen  by  reference  to  Figure  12,  Communications  Efficiency 
improved  within  and  across  training  conditions.  It  was  also  found  that 
variations  in  communications  efficiency  tended  to  result  from  changes  in 
recovery  difficulty  and,  in  fact,  to  be  sensitive  to  somewhat  finer  distinc¬ 
tions  between  difficulty  levels  than  appeared  to  be  the  case  with  the  pre¬ 
vious  measures. 
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Unfortunately,  no  firm  evaluations  can  be  made  regarding  transfer 
due  to  the  introduction  of  a  new  variable,  instructor  vs.  Bunker  Ramo 
judgment.  (Operational  performances  of  the  two  ships  may  have  differed 
as  a  function  of  this  variable  rather  than  arnounc  of  training  differences.) 

Based  on  the  operational  performance  curves  in  Figures  12  and  13, 
it  can  be  said  that  operational  communications  efficiency  was  judged  to 
be  more  than  adequate  even  though  some  f  these  recoveries  were  per¬ 
formed  v/ith  only  one  primary  approach  frequency.  And  the  following  can 
be  noted  regarding  opeiational  performance  of  individual  subteams:  (1) 
the  Ship  One  subteam  receiving  the  lowest  operational  rating  (3.7)  was  one 
including  an  “untrained"  approach  controller.  The  lower  rating  resulted 
from  the  judgment  that  this  member  of  the  subteam  communicated  more 
than  necessary  or  desirable,  (2)  The  Ship  Two  subteam  receiving  the 
lower  ratings  (3.  5  to  5.  0)  was  the  one  which  had  not  trained  in  the  device 
as  a  subteam  and  in  which  the  approach  controller  had  had  the  least  amt  ><  t 
of  training  (one  week).  This  controller  was  faulted  by  some  pilots  for  not 
communicating  enough  information.  So  the  evidence  does  suggest  that 
operational  subteam  communications  efficiency  is  better  if  device  training 
has  been  received,  even  though  it  is  not  conclusive. 

TEAM,  SUBTEAM  MEASURE:  COMMUNICATIONS  NOISE  LEVEL 
Defined  as  the  level  of  noise  generated  in  the  subteam  or  team  working 
area  due  to  communication  which  is  beyond  that  level  generated  by  normal 
use  of  the  primary  approach  frequencies.  Again,  a  graphic  rating  scale 
was  used  with  a  standard  of  comparison  as  described  above  for  communi¬ 
cations  efficiency.  Data  collected  on  the  Ship  Two  crew  assessed  subteam 
noise  level.  Due  to  developments  of  somewhat  different  coordination  pro¬ 
cedures,  the  Ship  One  crew  was  best  evaluated  at  the  team  level  on  this 
measure  during  the  first  week  of  training.  The  team  ratings  were  made 
entirely  by  instructor  personnel. 


I  The  same  comments  made  for  Communications  Efficiency  improve - 

I  ment  within  and  across  training  conditions  and  apparent  senskivitv  to 

5  recovery  difficulty  can  be  made  for  Communications  Noise  Level  (see 

!.  Figures  14  and  15),  The  only  deviant  curve  appears  in  Figure  14  and 

■  describes  the  second  week  of  training  for  one  subteam  from  Ship  One. 

1  It  is  possible  that  this  curve,  if  reflecting  accurate  judgments,  may  be 

primarily  a  function  of  situational  stresses,  partially  described  by  re- 
i  covery  difficulty  levels. 
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Again,  as  with  Communications  Efficiency,  no  firm  conclusions 
can  be  drawn  regarding  transfer  of  training.  The  same  comments  made 
for  Figures  12  and  I3can  be  made  for  Figures  14  and  15.  Again,  opera¬ 
tional  noise  level  was  very  good  on  Ship  Two,  with  the  subteam  which  had 
not  trained  as  a  subteam  receiving  the  lower  ratings  (4.  0  to  5.  0).  All 
the  Ship  One  subteams  received  adequate  but  lower  ratings,  reflecting 
the  higher  noise  levels  heard  on  this  ship. 

INDIVIDUAL  MEASURE  (APPROACH,  FINAL):  RADAR  CONTACT  NOT 
ANNOUNCED  TO  PILOT  Defined  as  the  percent  ige  of  the  total  number  of 
radar  contacts  that  should  have  been  announced  to  pilots  which  were  not 
annou  iced;  i.  e.  ,  it  is  an  error  measure  for  a  standard  communications 
procedure.  Communications  and  equipment  utilization  procedures  are 
trained  in  the  device  to  the  extent  inadequate  performance  of  these  pro¬ 
cedures  is  observed  by  the  instructors;  therefore  the  amount  and  type  of 
procedural  training  given  in  the  device  is  a  function  of  crew  requirements 
for  such  training.  An  opportunity  for  evaluating  the  value  of  the  training 
device  for  procedural  training  arose  when  instructors  observed  that  the 
Ship  Two  teams  (the  first  teams  observed  during  this  investigation)  were 
not  always  announcing  radar  contacts  to  the  pilots.  They  suggested  that 
a  measure  of  this  behavior  be  collected.  The  improvement  in  procedural 
communications  performance  on  the  part  of  individual  controllers  from 
both  ships  can  be  seen  in  Figure  16.  Operational  performance  by  indi¬ 
vidual  controllers  on  Ship  One  was  error  free.  Ship  Two  controllers 
purposely  did  not  announce  radar  contact  to  the  pilots  due  to  poorly  func¬ 
tioning  radars  during  the  observed  recoveries;  this  represented  a  con¬ 
sciously  executed  and  correct  performance  of  this  procedure. 

INDIVIDUAL  MEASURE  (APPROACH,  FINAL):  USE  OF  PRIMARY 
APPROACH  FREQUENCY  WITHIN  1/2  MILE  OF  RAMP  Use  of  this 
frequency  by  either  the  Approach  or  Final  controller  when  one  of  their 
aircraft  is  traversing  the  interval  between  1/2  mile  out  and  the  ramp  is 
to  be  avoided  due  to  possible  conflict  with  LSO  communications.  Use  of 
the  channels  during  this  interval  was  relatively  infrequent  and,  with  one 
exception,  the  Approach  controller  was  the  party  responsible.  Use  was 
generally  the  result  of  situational  necessity.  It  would  appear  from  the 
data  that  the  observed  controllers  understood  and  were  able  to  perform 
with  respect  to  this  criterion  acceptably  from  the  start  of  device  train¬ 
ing,  and  subsequently,  in  the  operational  environment.  As  no  trends 
were  apparent,  the  data  is  not  displayed  graphically.  In  terms  of  device 
evaluation,  it  can  only  be  said  that  the  device  provides  the  instructors 
with  an  opportunity  to  monitor  students  performance  with  respect  to  this 
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aspect  of  communications  behavior.  It  would  therefore  be  expected 
that,  as  was  the  case  with  the  previous  measure,  if  individual  control¬ 
lers  are  observed  using  the  primary  approach  frequency  unnecessarily 
during  this  interval  then  the  device  provides  an  opportunity  for  correc¬ 
tive  training. 

INDIVIDUAL  MEASURE  (APPROACH):  NUMBER  OF  CONFLICTIONS 
A  confliction  has  occurred  when  aircraft  come  within  unacceptably  close 
distances  of  one  another,  whether  or  not  they  actually  collide.  This  is 
an  especially  critical  measure  -  in  terms  of  both  life  and  equipment 
costs  -  but  a  difficult  one  to  reasonably  display  or  discuss  quantitatively 
due  to  its  seldom  occurrance  (unless  much  larger  data  samples  are  avail¬ 
able  than  obtained  in  this  study).  Instead,  textual  descriptions  of  the 
observed  conflictions  are  presented  below  to  provide  the  information 
needed  for  evaluation  purposes. 

a.  Trainer  Recoveries 

(1)  SHIP  ONE:  One  confliction  was  caused  by  both  Approach 
controllers  during  the  first  week  of  training.  One  controller 
froze  (didn't  communicate  with  either  the  pilot  or  the  other 
approach  controller)  while  the  other  controller  made  poor 
control  decisions  when  the  confliction  situation  began  develop¬ 
ing.  This  was  an  easy  recovery  using  two  minute  separations, 
but  it  was  the  third  recovery  using  NTDS  symbology.  The 
crew  did  not  attempt  to  use  the  symbology  again  during 
subsequent  training. 

(2)  SHIP  ONE:  One  confliction  was  caused  by  one  Approach 
controller  on  his  first  day  of  training  and  third  recovery  in 
this  position.  It  was  caused  by  poor  execution  of  the  inte¬ 
gration  function. 

(3)  SHIP  ONE:  Three  conflictions  on  two  recoveries  in  sequence 
on  the  same  day  were  caused  by  the  same  Approach  con¬ 
troller  during  his  second  week  of  device  training  (performance 
during  his  first  week  of  training  was  not  observed).  Decision 
was  made  not  to  use  this  person  as  a  controller  based  on  this 
and  other  aspects  of  performance. 

(4)  SHIP  TWO:  Two  conflictions  in  separate  and  difficult  prob¬ 
lems  were  caused  by  the  same  controller.  Other  device 
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and  operational  measures  of  performance  on  this  controller 
indicate  he  is  one  of  the  better  Approach  controllers.  Of 
relevance  here  is  his  expressed  opinion  that  the  trainer  is 
the  place  to  test  capability  limits  and  to  try  out  alternative 
solutions,  "because  in  the  trainer  you  won't  kill  anybody". 

b.  Ship  Recoveries 

(1)  SHIP  ONE:  Appears  to  have  been  one  or  two  conilictions 
{or  at  least  situations  which  were  dangerously  close  in  both 
the  pilot  and  another  controller's  opinion).  The  faulted  con¬ 
troller  did  not  receive  approach  control  training  in  the  device. 

(2)  SHIP  TWO:  One  confliction,  caused  primarily  by  intermit¬ 
tently  operating  radar.  It  appears,  however,  that  the  situa¬ 
tion  may  have  been  initially  set  up  by  the  Approach  controller 
who  had  received  only  one  week  of  device  training. 

Based  on  the  above  descriptions  and  the  extreme  criticality  of  this 
measure,  it  appears  the  following  should  be  said:  (1)  Inexperience  can 
lead  to  conflictions  in  the  training  device  (see  a.  (1)  and  (2)  above),  (2) 

If  training  device  performance  and  behavior  is  similar  to  operational 
performance  and  behavior  (and  the  objective  data  and  observations  support 
this),  then  personnel  evaluation  and  selection  based  on  training  device 
performance  will  reduce  the  likelihood  of  conflictions  during  actual  CATCC 
controlled  recoveries  (a.  (3)),  (3)  The  device  may  be  valuable,  not  only 

for  developing  proficiency,  but  also  for  evaluating  limits  and  alternative 
solutions  (a.  (4)),  and  (4)  The  likelihood  of  conflictions  in  the  operational 
environment  is  affected  by  training  in  the  CATCC  training  device  (b.  (1) 
and  (2)). 

INDIVIDUAL  MEASURE  (APPROACH  CONTROLLER  IN  SUBTEAM  A): 
DEVIATION  FROM  SCHEDULE  RAMP  TIME  FOR  FIRST  AIRCRAFT 
Defined  as  the  difference,  in  minutes,  between  the  time  the  first  aircraft 
reaches  the  ramp  and  the  time  the  ramp  is  scheduled  to  be  open.  Measured 
whenever  the  time  of  arrival  can  be  attributed  to  the  Letdown  controller 
rather  than  other  events  (e.  g.  ,  Marshaller  error,  unannounced  heading 
change  just  prior  to  ramp  time). 

Based  on  the  curves  presented  in  Figure  17,  it  must  be  concluded 
that  Ship  One  Approach  controllers  did  not  improve  their  performance  on 
this  measure  while  in  the  training  device  (other  Ship  One  data,  not 


f  .gure  17.  Individual  Measure  (letdown  A):  Deviation  From 
Scheduled  Ramp  Time  for  First  Aircraft 
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included  in  Figure  17  due  to  incomplete  samples,  also  failed  to  indicate 
learning).  In  contrast  to  Ship  One,  data  for  Ship  Two  Approach  controllers 
(again,  cannot  be  presented  graphically  due  to  incomplete  samples)  de¬ 
scribe  perfect  performances  (zero  deviations)  on  five  of  the  seven  mea¬ 
surable  recoveries.  Of  the  two  missed  ramp  times,  one  was  controlled 
by  a  controller  in  the  Approach  position  for  the  first  time  (1.5  minute,  miss) 
while  the  other  was  confounded  by  situational  events.  The  operational 
data  describe  the  Ship  One  controller  who  did  not  receive  training  in  that 
position  in  the  device  and  the  Ship  Two  controller  who  had  received  only 
one  week  of  device  training. 

It  is  unclear  from  the  above  discussion  whether  or  not  the  training 
device  is  effective  for  training  this  aspect  of  performance.  Based  on 
observation,  overall  similarities  between  recoveries,  and  discussion 
with  instructor  personnel,  it  is  suggested  that  Figure  17  is  indicative  of 
device  utilization,  rather  than  device  capability.  CATCC  crews  are 
given  classroom  instruction  in  the  importance  of  meeting  ramp  times 
and  how  to  meet  this  criterion.  Instructor  personnel  indicate  that  if  ramp 
times  are  missed  to  any  extent  or  consistently  this  is  pointed  out  to  the 
controllers  and  corrective  instruction  given.  In  the  case  of  Ship  One, 
however,  there  were  a  number  of  other  personnel  and  coordination  pro¬ 
blems  of  much  greater  importance  which  required  resolution  insofar  as 
possible  during  the  training  period.  In  other  words,  it  is  suggested  that 
performance  by  Ship  Two  Letdown  A  controllers  represent  training  device 
capability,  while  performance  by  Ship  One  Letdown  A  controllers  repre¬ 
sent  a  case  of  training  device  utilization. 

INDIVIDUAL  MEASURE  (MARSHALLER):  MARSHAL  INFORMATION 
BROADCAST:  COMPLETENESS  AND  ACCURACY  During  the  marshall¬ 
ing  period,  the  marshaller  is  responsible  for  communicating  a  standard 
set  of  informational  items  to  each  of  the  pilots.  This  is  accomplished 
both  by  communications  to  individual  pilots  and  by  general  information 
braodcasts.  Ship  One  Marsha’.lers  were  monitored  periodically,  especially 
during  the  first  week  of  training.  No  errors  ( 100%  accuracy)  or  omissions 
(100%  complete)  were  observed.  Due  to  manpower  constraints,  data  were 
not  collected  on  this  measure  on  board  Ship  One.  A  Ship  Two  Marshaller 
made  one  error  on  the  first  day  of  training  which  complicated  that  re¬ 
covery.  From  that  point  on,  no  omissions  and  no  further  errors  were 
observed.  Data  collected  on  board  Ship  Two  indicates  100  percent  accuracy 
and  completeness  for  Marshaller  broadcasts. 
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The  conclusion  regarding  trainer  utility  is  that  it  provides  an 
opportunity  for  corrective  instruction  if  such  is  needed.  If  a  broadcast 
error  or  omission  is  made,  the  students  are  provided  with  an  opportunity 
to  realize  the  possible  effects  of  such  errors  and  omissions.  The  pri¬ 
mary  value  of  the  trainer  for  this  activity  appeared  to  be  the  opportunity 
it  provided  the  Marshallers  and  Marshal  Status  Board  Keepers  for  de¬ 
veloping  coordinated  procedures  which  would  avoid  errors  and  omissions. 
Broaacast  errors  and  omissions  were  made,  especially  early  in  training, 
but  were  not  scored  as  such  because  they  were  caught  and  corrected  by 
these  personnel.  It  was  observed  that,  as  a  result  of  these  errors  and 
omissions,  procedures  were  purposely  developed  by  the  students  to  avoid 
their  recurrance. 

INDIVIDUAL  MEASURE  (MARSHALLER):  EAT  DEVIATIONS  BEYOND 
EAT  *.  25  MINUTES  NOT  ACTED  ON  Defined  as  the  percentage  of 
incidences  not  acted  upon  by  the  Marshaller.  An  incident  occurred  when¬ 
ever  a  pilot  did  not  depart  the  marshalling  area  within  i\  25  minutes  of 
his  assigned  EAT.  As  an  incident  did  not  occur  in  every  recovery,  only 
those  recoveries  having  such  events  were  considered  for  this  measure 
(i.  e.  ,  Trainer  Recovery  No.  3  in  Figure  18  is  the  third  recovery  in  which 
each  of  the  Marshallers  experienced  this  type  of  problem). 

It  does  not  appear,  based  on  the  curves  presented  in  Figure  18,  that 
the  student  Marshallers  tended  to  improve  this  aspect  of  their  perfor¬ 
mance  as  a  result  of  trainer  use.  The  only  student  Marshaller  exhibiting 
what  might  have  been  a  learning  trend  was  the  Ship  One  student  Marshaller 
who  later  performed  in  this  position  onboard  the  carrier  (see  Figure  18, 
Individual  Performance  Curve,  recoveries  2-4).  This  Marshaller  did  act 
correctly  on  the  one  incident  occurring  during  observed  Ship  One  opera¬ 
tional  recoveries. 

The  Ship  Two  Marshaller  who  performed  in  this  posistion  during 
operational  recoveries  used  a  self -defined  criterion  of  EATi'l.  00  minute 
during  trainer  recoveries;  corrective  instruction  was  not  given.  What 
EAT  criterion  this  marshaller  used  onboard  is  not  known;  observation 
of  operational  behavior  and  interviews  indicated,  however,  that  he  had 
become  aware  of  time  as  an  important  criterion  with  respect  to  Marshal 
control  activities. 

Based  on  the  data,  it  appears  that  one  Marshaller  may  have  learned 
to  perform  with  respect  to  the  EAT  criterion  as  a  result  of. device  train¬ 
ing.  None  of  the  other  Marahallers  exhibited  any  consistent  improvement 
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while  in  the  device.  From  this  evidence  it  must  be  concluded  that 
positive  effects  on  this  aspect  of  Marshaller  performance  as  a  result  of 
device  training  appear  doubtful  and  it  should  be  questioned  whether  thi'. 
apparent  lack  of  consistent  performance  improvement  is  the  result  of 
training  device  capability  or  utilization.  Based  on  a  review  of  device 
capabilities,  student  responses  to  queries  regarding  the  EAT  criterion, 
and  observation,  it  appears  that  Figure  18  reflects  training  device  utili¬ 
zation  and  the  following  suggestions  can  be  made: 


with  PC  &E  program  time.  Some  student  marshallcrs  indicated 
a  reluctance  to  take  action  on  any  but  extreme  EAT  deviations 
due  to  suspicions  that  PC  E  time  was  not  synchronized  with 
their  mission  time.  The  result  was  that  even  extreme  deviations 
were  missed.  These  suspicions  arose  when  aircraft  consistently 
departed  Marshal  with  an  EAT  deviation  of  similar  magnitude. 

This  suggestion  can  be  easily  implemented  by  more  careful  setting 
of  the  mission  clocks  (these  are  reset  at  the  start  of  each  trainer 
recovery). 

b.  Use  EAT  deviations  on  a  more  systematic  basis  as  determined 
by  student  performance.  The  time  of  departure  from  marshal 
is  controlled  by  PC  &  E  pilots  who  are  working  under  instructor 
supervision.  It  appears  from  the  collected  data  that  EAT  devia¬ 
tions  occurred  randomly;  that  is,  the  were  not  used  as  an  instruc¬ 
tional  tool  for  either  this  aspect  of  Marshaller  performance  or 
the  development  of  coordinated  team  procedures  for  EAT  devia¬ 
tions.  It  is  suggested  that  the  Marshaller's  EAT  devia!  on 
behavior  be  monitored  more  closely  and  that  deviations  b 
inserted  into  the  problems  as  needed  to  train  Marshaller  and 
team  r  -formance  to  acceptable  levels. 

c.  Provide  corrective  instruction  when  EAT  deviations  are  not 
acted  upen.  As  this  is  not  on'  of  the  more  important  training 
objective:--  (conducting  a  safe  recovery  under  difficult  conditions 
is  the  primary  training  objective)  and  EAT  deviations  can  dis¬ 
organize  recoveries  to  such  an  extent  that  they  become  poor 
vehicles  for  the  other  training  objectives,  it  is  important  that 
EAT  deviations  not  be  used  more  than  is  realistic  or  is  necessary 
for  Marshaller  and  team  coordination  training.  The  instructional 
effect  of  EAT  deviations  w*ll  be  increase'4  (so  that  fewer  need  be 
used)  to  the  extent  that  corrective  instruction  is  given  by  the  in¬ 
structors  when  the  student  Marshaller  fails  to  act  appropriately. 
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EVALUATIVE  PERFORMANCE  MEASURES  It  will  be  retailed  that  I 

evaluative  performance  data  were  collected  from  student  controllers,  I 

pilots,  LSOs  and  the  ORI  team.  The  purposes  of  these  data  were  both  tr>  j 

establish  the  adequacy  of  contro’ler  performance  during  operational  re-  j 

coveries  and  to  provide  explanatory  background  information  which  would  j 

assist  data  interpretation  and  device  evaluation  efforts.  j 


CONTROLLER  SELF-EVALUATIONS  The  student  controllers  were  J 

asked  to  evaluate  their  own  performance  with  respect  to  Communications,  I 

Guidance,  Emergency  Procedures,  Contingency  Handling,  and  Overall  j 

j 

Performance  using  three  and  five  point  scales.  They  made  the  same  set  ; 

of  evaluations  twice,  once  within  the  training  device  and  once  onboard 
the  carriers.  The  evaluations  were  made  in  the  trainer  at  the  midpoint  j 

of  the  observed  training  periods.  At  sea  the  evaluations  were  made  be-  j 

tween  or  after  observed  recoveries.  The  results  presented  in  Figure  19  1 

indicate  that  the  controllers  generally  evaulated  themselves  as  performing 
at  Good  Excellent  levels  during  operational  recoveries  and  that  their  opera¬ 
tional  performance  was  judged  to  be  better  than  their  trainer  performance. 

These  results  are  as  one  would  predict  if  the  trainer  is  hypothesized  to 
provide  rigorous  training  which  prepares  the  student  to  perform  adequately 
in  the  operational  environment. 


It  is  of  interest  that  the  only  two  individual  operational  performance 
evaluations  which  were  lower  than  their  trainer  evaluations  were  both 
made  by  comparatively  inexperienced  and  young  controllers  (Approach 
and  Marshal  positions)  in  the  Emergency  Procedures  category.  As  stated 
earlier,  instructor  personnel  feel  that  their  most  important  training 
objective  is  to  prepare  teams  to  conduct  safe  recoveries  under  difficult 
conditions.  Cne  of  the  frequently  used  problem  variables  is  that  of  air¬ 
craft  emergencies  and  contingencies.  The  responses  by  these  controllers, 
in  particular,  indicates  that  encountering  these  problems  during  actual 
recoveries  is  indeed  difficult  and  requires  preparation. 

It  is  also  of  interest  to  note  thnl  the  greatest  difference  between 
Trainer  a(  d  Ship  self-evaluations  takes  plac-  in  the  Guidance  category. 
This  difference  will  be  relevant  to  the  discussions  of  evaluative  training 
device  measures  (pp.  81-82). 

PILOT  EVALUATIONS  OF  CATCC  PERFORMANCE  Pi’ots  participating 
in  the  observed  operatic  nal  recoveries  were  requested  to  complete  an 
evaluation  from  for  the  Marshal,  Appioach,  and  Final  controllers  with 
respect  to  Communications,  Guidance,  Emergency  Procedures,  Contin¬ 
gency  Han  ling ,  and  Overall  Performance  using  a  three  point  scale.  The 
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Figure  19.  Controller  Self-Evaluation: 
and  Ship  Performance  Evaluations 
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form  also  allowed  evaluation  of  overall  team  performance  on  a  five 
point  scale  and  provided  an  opportunity  for  comments,  explanations  and 
suggestions.  As  indicated  by  the  results  in  Table  5,  the  controller  3  in 
each  of  the  positions  were  judged  between  "good"  and  "excellent"  on  each 
of  the  tasks  by  the  pilots  under  their  control.  It  will  be  noted  that  Guid¬ 
ance  received  the  lowest  evaluations;  again,  this  will  be  discussed  in 
greater  detail  later  (pp.  81-82).  The  mean  team  rating  was  3.  8  (betv/een 
"average"  and  "excellent")  which  indicates  the  pilots  felt  that  the  overall 
team  performance  was  also  quite  adequate. 

LSO  EVALUATIONS  OF  CATCC  PERFORMANCE  LSOs  who  performed 
during  the  observed  recoveries  were  interviewed.  CATCC  performance 
was  judged  adequate  by  these  personnel  on  both  Ships  One  and  Two. 

ORI  EVALUATIONS  OF  CATCC  PERFORMANCE  The  results 

of  GRP,  conducted  on  both  Ships  One  and  Two  indicate  quite  adequate 
performances  in  both  instances. 

EVALUATIVE  TRAINING  DEVICE  MEASURES  Student  controllers  and 
instructor  personnel  completed  training  device  questionnaires  which  re¬ 
quired  device  evaluation  from  three  standpoints:  (1)  Task  environment 
realism,  (2)  Task  performance  realism,  and  (3)  Training  effectiveness 
(see  Figures  20  through  22).  Other  questions  were  asked  regarding  NTDS 
symbology  and  additional  effective  or  ineffective  characteristics. 

The  rating  responses  to  questionnaire  items  are  displayed  in  Figures 
20  through  22  for  five  populations:  (1)  Ship  One  student  responses  during 
training,  (2)  Ship  Two  status  board  keepers  during  training  (responses 
made  by  this  crew's  board  keepers  were  distinctly  different  in  many  cases 
from  those  made  by  the  controllers,  making  separate  displays  advisable), 
(3)  Ship  Two  student  controllers  daring  training,  (4)  Ship  Two  student 
controllers  during  onboard  operations,  and  (5)  Instructor  personnel. 

To  provide  an  overall  perspective,  three  general  data  trends  can  be 
pointed  out: 

a.  The  CATCC  training  device  generally  received  moderate  to 
high  ratings  on  both  realism  and  effectiveness  (Figures  20-22 
and  Table  6). 

b.  The  CATCC  training  device  generally  received  higher  ratings 
on  effectiveness  than  on  realism  (Figures  2 0-22  and  Table  6). 
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TABLE  5.  PILOT  EVALUATIONS  OF  CATCC 
PERFORMANCE 


PERFORMANCE 

CATCC  POSITION 

MEAN 

VARIABLES 

MARSHAL 

APPROACH 

FINAL 

EVALUATION 

Communication 

2.  2* 

2.  1 

2.  1 

2.  1 

Guidance 

2. 

2.  0 

1.9 

2.  0 

Emergency 

Procedures 

2.  6 

2.  3 

2.  2 

2.  4 

Contingency 

Handling 

2.  1 

2.  1 

2.  4 

2.  2 

Overall  Performance 

2.  2 

i.  0 

2.  0 

2.  1 

Mean 

Evaluation 

2.  2 

2.  1 

1 

2.  1 

_ 1 

*  Values  of  1,2,  or  3  assigned  to  pilot  selection  of  Poor,  Good  or 
Excellent  on  a  mating  scale. 
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Figure  20,  Trainer  Evaluations:  Task  Environment  Realism 
(N  =  11,  3,  4,  4,  4  ) 


;^s«rTTOf»H^'.  _  - 


»  •vjiwzftr*  t  - 


NAVTRADEVCEN  70-C-0258-1 


COMMUNICATION 


NOTE:  See  previous 
figure  for 
legend. 


1 


GUIDANCE 

CONTROL 


CONTINGENCY 

HANDLING 


EMERGENCY 

SITUATION 


STUDENT 

PERFORMANCE 


•  *  i'\^‘  •*  •  •*«•  • » •**•*•»•»*•*«  •  •  *•»•  *••  ••  ®% ••  i***>  »"«*,**•  ,**«  * *1  *• 


mMmammmMi 


A 

Low 

Effectiveness 


A 

High 

Effectiveness 


Figure  21.  Trainer  Evaluations:  Task  Performance  Realism 
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TABLE  6 

Average  Responses  to  Questionnaire  Items 
Related  to  Trainer  Realism  vs.  Effectiveness  Characteristics 


TRAINING 

EFFECTIVENESS 


i? 

i 

t 

TRAINER  REALISM 

.  V 

Task 

Environment 

l  Task 

|  Performance 

'f  S' 

;  i. 

Student  Controllers 

3.  2* 

J  3.4 

i  * 

:  p 

v  i' 

Instructor  Personnel 

4.  0 

|  3.9 

♦Mean  response  using  a  five  point  scale  where  1  =  Low  Effectiveness 
or  Low  Realism  and  5  =  High  Effectiveness  or  High  Realism. 


TABLE  7 

Comparison  of  Questionnaire  Responses 
Made  During  Training  vs.  Operations 


T  raining 
Operations 


TRAINER  REALISM 


Task 


Task 


Environment  \  Performance 


TRAINING 

EFFECTIVENESS 


3.4  |  3.5 

3.  7 

1 

2.  9  *  3.  4 

4.  0 

1 

. . .  < 


c.  When  student  controllers  reevaluated  the  trainer  after  having 
controlled  actual  recoveries  they  found  the  device  less  real, 
but  more  effective.  Specifically,  they  gave  lower  evaluations 
of  environmental  realism,  similar  evaluations  of  task  perfor¬ 
mance  realism,  and  higher  evaluations  of  training  effectiveness 
(Figures  241-22  and  Table  7). 

The  most  essential  of  these  questions  in  terms  of  trainer  effective¬ 
ness  evaluation  are  those  dealing  with  training  effectiveness.  Questions 
concerning  environment  and  performance  realism  are  important  to  the 
extent  they  provide  information  on  how  training  effectiveness  might  be 
improved;  this  information  is  particularly  important  if  it  can  be  related 
to  training  weaknesses  identified  by  the  objective  or  evaluative  data.  On 
the  whole,  both  objective  and  evaluative  performance  data  have  indicated 
the  CATCC  trainer  to  be  quite  effective.  The  general  trends  of  the  device 
evaluative  rating  data  presented  above  describe  the  same  picture.  It  is 
also  true,  however,  that  some  of  these  data  and  certain  observations 
indicate  that  device  training  effectiveness  could  be  improved.  Student 
comments  and  suggestions,  made  in  conjunction  with  their  ratings,  are 
relevant  to  these  considerations  and  are  discussed  in  the  following 
paragraphs. 

TASK  ENVIRONMENT  REALISM:  TARGETS  AND  OTHER  RAD/  *.  PI? 
PLAY  MARKINGS  All  responses  to  this  question  criticized  t.  °  »*niar 
displays  on  one  or  both  of  two  counts:  (1)  Targets  are  too  large  .d  dis¬ 
tinct,  and  (2)  Target  displays  are  too  persistent  (i.  e.  ,  do  not  "fade"). 

It  is  interesting  to  note  that  CATCC  crews  from  the  Atlantic  fleet  ct  ng 
to  the  training  device  make  just  the  opposite  criticisms;  ie.  e.  ,  that  the 
display  quality  is  too  poor  and  not  persistent  enough.  Device  radar  dis¬ 
plays  simulate  displays  which  re  intermediate  in  distinctness  and  per¬ 
sistency  between  those  found  in  the  Pacific  vs.  the  Atlantic  fleets.  As 
the  device  must  serve  both  fleets,  and  there  is  no  evidence  that  training 
effectiveness  was  degraded  by  this  lack  of  complete  realism,  this  appears 
to  be  a  reasonable  decision. 

TASK  ENVIRONMENT  REALISM:  RADAR  DISPLAY  BEHAVIOR  Target 
behavior  within  a  ten  mile  radius  of  the  ship  was  strongly  criticized. 
Further,  it  is  apparent  from  tl  *  associated  comments  that  some  of  the 
lower  ratings  given  the  item  above  reflected  controller  opinion  regarding 
target  behavior  (rather  than  display  realism,  the  question  asked).  The 
criticisms  were  related  to  target  movement  (spasmodic,  turns  too  large) 
and  to  target  response  to  control  communications  (time  lag  too  long). 


-Ti 1  V  ^*.17 


P  5WT^  TfST^V? 


NAVTRADE VCEN  70-C-0258-1 


The  lower  ratings  given  this  realism  characteristic  are  important  in 
that  some  of  the  associated  comments  indicated  that  it  may  have  affected 
the  controllers'  guidance  behavior  in  the  trainer  (e.  g.  ,  "...  due  to  large 
turns  and  at  times  extreme  control  required,  realism  fades  to  non¬ 
existent  at  times",  "Targets  behave  realistically  when  the  program  is 
running  good,  Otherwise,  forget  it.")  Further,  it  will  be  recalled  (p.  2-4) 
that  the  controllers  evaluated  their  guidance  performance  as  poorer  in 
the  trainer  but  as  quite  adequate  during  operational  recoveries;  while 
pilots  gave  their  lowest  evaluations  of  controller  performance  in  the 
guidance  category.  Based  on  these  evidences,  it  is  suggested  that  the 
training  device  is  not  entirely  adequate  in  training  guidance  behaviors  and 
that  the  this  inadequacy  is  caused  by  unreal  target  movement  and  response 
to  controller  communications  within  the  ten  mile  radius.  It  is  suggested 
that  students  may  not  be  learning  acceptable  guidance  criteria  (i.  e.  ,  may 
be  accepting  larger  flight  path  and  pattern  deviations  during  operations 
than  are  necessary)  and  that  development  of  optimal  guidance  control 
behaviors  in  training  may  be  difficult  where  target  response  is  not  fairly 
realistic  or  consistent. 


There  are  two  actions  that  can  be  taken  which  would  improve  the 
realism  of  target  behaviors,  one  of  which  is  currently  being  implemented: 
(1)  Develop  software  programs  which  will  cause  more  realistic  target 
movements  (such  programs  are  presently  in  development  and  expected 
to  be  in  operation  soon)  and  (2)  Use  only  those  personnel  who  can  meet 
acceptable  performance  standards  to  fill  pilot  positions  in  PC&E.  The 
latter  action  relates  to  the  response  time  lag  criticisms  and  to  observations 
of  controller  irritation  and  confusion  on  those  occasions  where  pilots  re¬ 
sponded  incorrectly.  As  trainer  utilization  increases  (and  it  does  appear 
to  be  increasing),  it  will  become  easier  to  maintain  a  staff  of  trained 
pilot  personnel,  thus  implementing  this  action  to  some  extent.  Complete 
implementation  may  require  more  formal  pilot  training  and  qualification 
procedures  than  are  currently  in  effect. 

TASK  ENVIRONMENT  REALISM:  SITUATIONAL  RESPONSE  TO  STUDENT 
PERFORMANCE  Few  critical  comments  were  made  to  this  question  and 
none  that  suggested  training  device  modification  (e.  g.  ,  "Bolter  and  wavc- 
off  rate  too  high"-  whicn,  incidontly,  is  a  problem  variable  purposely  used 
to  develop  team  capability  to  handle  a  high  rate;  "A  longer  time  for  train¬ 
ing").  It  can  also  be  noted  that  the  ratings  on  this  environmental  realism 
characteristic  tended  to  be  higher  than  on  the  previous  two  lvigure  20). 
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TASK  ENVIRONMENT  REALISM:  TRAINING  PROBLEMS  A  review 
of  comments  made  on  thij  item  and  on  an  item  concerning  the  training 
effectiveness  of  the  problems  suggests  that  controllers  with  actual 
CATCC  experience  tend  to  feel  that  real  problems  are  being  simulated: 
"Problems  equal  to  if  not  greater  than  problems  encountered  in  the  fleet", 
"Overall  presentation  of  problems  are  a  close  parallel  to  actu;l  opera¬ 
tions",  "Problems  should  be  more  complicated".  In  contrast,  controllers 
without  previous  CATCC  experience  tend  to  feel  that  operational  recover¬ 
ies  can't  be  that  difficult  and  such  problems  don't  really  exist.  "You 
wouldn't  find  that  many  problems  in  a  recovery",  "There  are  too  many 
off  the  wall  conditions  thrown  in  that  do  not  exist",  "Things  go  a  bit 
smoother  (during  operations)".  With  respect  to  the  foregoing  comment 
re:  nonexistent  conditions,  an  instructor  response  to  the  same  question 
is  pertinent:  "We  use  actual  situations  we  have  been  involved  in.  "  With 
respect  to  the  last  critici.-m  re:  smoother  actual  operations,  a  comment 
made  by  this  same  control.  '-  on  his  self-evaluation  form  after  an  opera¬ 
tional  recovery  is  of  inter e.  "It  will  all  work  out  (I  hope!)".  In  sum¬ 
mary,  the  training  problems  appear  to  be  realistic  and  to  serve  training 
objectives. 


TASK  PERFORMANCE  REALISM:  COMMUNICATION  The  questionnaire 
did  not  request  comments  on  ratings  of  task  performance  realism  (Ques¬ 
tion  12).  However,  comments  made  in  response  to  other  questions  pro¬ 
vide  a  basis  for  a  discussion  on  communications  performance  realism  and 
suggest  the  need  for  such  discussion.  As  the  communications  channels 
set  up  in  the  training  device  are  the  same  as  found  on  Ship  Two,  Ship  Two 
controllers  had  little  difficulty  accepting  the  realism  of  the  communica¬ 
tions  simulation  (Figure  21).  Trainer  communications  realism  was  down¬ 
graded  after  onboard  experience  due  to  serious  problems  with  the  on¬ 
board  communications  equipment  (cf.  ,  Figure  21). 


Ship  One  controllers  did  not,  however,  accept  the  trainer  communi¬ 
cations  set  up  as  a  completely  real  one.  As  the  training  device  is  not 
presently  equipped  with  a  CATCC  intercommunication  system  (ICS),  crews 
from  ships  having  this  equipment  (e.  g.  ,  Ship  One)  cannot  integrate  its 
usage  into  the  coordination  procedures  they  develop  during  device  tracing. 
Thu  effect  of  this  lack  of  device  simulation  capability  is  reflected  in  the 
lower  ratings  given  to  Communications  Performance.  Realism  (Figure  21) 
and  Crew  Coordination  Development  Training  Effectiveness  (Figure  22) 
given  by  the  Ship  Or.e  crew. 


Although  the  actual  impact  of  ICS  vs.  no  ICS  conditions  on  coordina¬ 
tion  procedures  and  communications  performance  may  not  be  extensive 
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and,  therefore,  may  not  have  to  be  considered  for  training  purposes, 
it  is  generally  felt  that  training  device  acceptance  enhances  training 
device  effectiveness.  Although  it  may  be  advisable  to  continue  to  train 
the  crews  from  all  ships  to  perform  without  use  of  the  ICS  (in  prepara¬ 
tion  for  degraded  or  failed  communications  equipment  conditions),  it  is 
suggested  that  availability  of  ICS  equipment  in  the  CATCC  training  device 
for  use  by  crews  from  ICS-equipped  ships  will  increase  controller  accep¬ 
tance  and,  as  a  result,  may  increase  training  effectiveness. 

TRAINING  EFFECTIVENESS:  CREW  COORDINATION  DEVELOPMENT 
Again,  comments  were  not  requested  with  respect  to  this  rating  (Ques¬ 
tion  2),  but  responses  made  elsewhere  in  the  questionnaire  relate  to  this 
effectiveness  characteristic  and  will  be  discussed  here.  (Cr  ew  coordina¬ 
tion  in  terms  of  communications  procedures  is  discussed  in  the  above 
paragraph's. ) 

It  will  be  recalled  that  the  Approach  controller  performs  two  functions, 
control  and  integration  (see  p.  21  and  Figure  3).  There  are  a  few  CATCCs 
which  are  equipped  such  that  the  integration  function  can  be  performed  by 
a  separate  controller  (called  a  Bolter /Waveoff  controller)  to  whatever 
extent  the  CATCC  team  chooses.  The  CATCC  on  Ship  One  is  equipped  j 

in  this  manner  but  the  training  device  is  not  presently  configured  to  allow 
realistic  training  in  this  position  with  associated  development  of  crew 
coordination  procedures.  Although  the  Ship  One  Bolter /Wafeoff  controllers 
could  have  practiced  both  their  control  and  integration  skills  in  the  train¬ 
ing  device  as  it  is  presently  configured,  .  ey  did  not  do  so.  It  appears 
that  one  reason  they  did  not  do  so  may  have  been  their  judgment  that  crew 
coordination  skills  relevant  to  their  position  could  not  be  realistically 
developed  (Lack  of)  internal  communications  between  Bolter /Waveoff  and 
Letdown  controllers  (is  ineffective)11,  "The  couse  shoudl  be  able  to  dupli¬ 
cate  the  different  CCA  set-ups  aboard  ship.  We  are  unable  to  implement 
our  Bolter /Waveoff  position").  This  is  of  particular  importance  because 
the  evidence  suggests  that  the  conflictions  occurring  on  this  ship 
(p.  67  )  might  have  been  avoided  if  these  personnel  had  taken  advantage 
of  the  Approach  control  training  provided  by  the  trainer;  and  probably 
would  have  been  avoided  if  they  had  cither  trained  as  a  team  with  this 
position  or  had  made  a  clear  decision  not  to  implement  this  position  on¬ 
board. 

It  is  suggested  that  either  one  ol'  two  approaches  be  taken  to  provide 
training  to  crews  from  ships  equipped  for  the  Bolter /Waveoff  position: 

(1)  That  the  present  equipment  configuration  withm  the  CATCC  training 
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device  be  retained  and  that  communications  equipment  necessary  to  imple¬ 
ment  the  Bolter /Waveoff  position  be  added.  This  would  permit  the  position 
to  be  simulated  to  some  extent,  thus  allowing  skills  and  coordinations  pro¬ 
cedures  to  be  developed.  To  the  extent  control  and  integration  skills 
require  more  extensive  practice,  these  personnel  could  be  encouraged 
to  perform  in  the  Approach  control  position.  (2)  That  the  CATCC  train¬ 
ing  device  be  equipped  and  reconfigured  such  that  an  actual  Bolter /Waveoff 
position  is  available  for  those  crews  desiring  to  train  with  this  position. 

TRAINING  EFFECTIVENESS:  COMPARED  TO  ONBOARD  TRAINING 
All  the  comments  made  with  regard  to  this  question  suggest  the  same 
conclusion:  the  “real  thing"  provides  the  best  training  but  that  on-the-job 
training  is  entirely  too  dangerous.  (At  present,  the  only  feasible  onboard 
training  program  is  on-the-job  training.)  Further,  the  training  device 
has  the  effective  features  of  flexibility  and  instructional  opportunity.  A 
wide  range  of  possible  operational  problems  and  conditions  can  be  simula¬ 
ted,  providing  repeated  exposure  to  difficult  situations  that  require  co¬ 
ordinated  and  appropriate  responses.  The  trainer  is  also  flexible  in  terms 
of  the  load  conditions  that  can  be  imposed  on  a  student.  The  ability  to 
stop  a  recovery  problem  or  to  repeat  the  same  problem  provides  oppor¬ 
tunities  to  instruct  and  discuss  problem  areas,  and  or  procedures  to  be 
developed,  practiced  and  evaluated. 

NTDS  SYMBOLOGY  EFFECTIVENESS  Question  9  asked,  "How  effective 
is  the  symbology  in  aiding  the  controller  at  your  position?"  The  average 
rating  was  1.  4  (1.  0  =  Low  Effectiveness)  and  all  comments  lead  to  the 
same  conclusion:  Without  automatic  tracking  of  the  aircraft  by  the  sym¬ 
bols,  the  symbology  is  a  hindrance  and  possibly  a  danger;  with  automatic 
tracking,  it  is  feasible  but  the  controllers  question  whether  the  possible 
value  of  an  additional  information  source  is  worth  the  scope  clutter  and 
interference  with  the  "real"  video  world  caused  by  the  symbology.  The 
question  of  the  value  of  NTDS  symbology  to  CATCC  operations  may  be 
worth  investigating  in  more  detail. 

ADDITIONAL  EFFECTIVE  AND  INEFFECTIVE  TRAINING  DEVICE 
CHARACTERISTICS  The  majority  of  comments  made  in  response  to 
queatioi  naire  items  have  been  covered  by  the  above  discussions.  One 
additional  type  of  comment  was  made,  however,  which  does  not  relate 
to  training  device  effectiveness  characteristics  but  rather  to  training 
device  utilization.  This  type  of  comment  was  made  by  Ship  One  control¬ 
lers  during  their  first  week  of  training:  "Occasionally  allow  a  problem  to 
run  smoothly  with  little  or  no  changes  so  as  to  build  some  confidence  in 
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controllers.  With  every  problem  a  'flail'  one  begins  to  develop  doubts 
in  one’s  ability.  M  This  comment  was  made  in  spite  of  the  fact  that  the 
recovery  problems  performed  by  this  crew  were  generally  much  easier 
than  those  performed  by  the  Ship  Two  crev/s. 

Unfortunately,  there  are  presently  no  regulations  in  effect  which 
require  a  CATCC  crew  to  remain  in  training  until  a  determination  of 
"qualified"  is  made  by  instructor  personnel.  As  a  result,  instructors 
must  make  maximum  use  of  the  known  available  time  to  prepare  the  crew 
for  the  situations  that  can  arise  in  the  operational  environment.  The 
instructional  problem  is  compounded  by  the  empirical  facts  that  changes 
in  team  and  subteam  composition  tend  to  present  new  team  and  subteam 
learning  requirements.  These  compositional  changes  occur  throughout 
training.  Further,  new  requirements  for  individual  training  continue  to 
arise  throughout  training  whenever  cross-training  takes  place.  As  a 
result  of  these  time  constraints  and  learning  requirement  multiplications, 
instructors  must  compi  omise  the  development  of  confidence  and  full  pro¬ 
ficiency  to  ensure  the  development  of  crew  coordination  procedures  and 
implementation  methods  adp-juate  for  the  operational  situations  that  can 
and  do  arise.  It  is  suggested  that  CATCC  qualification  procedures  and 
standards  be  developed  and  that  qualification  requirements  be  established, 
so  that  the  full  potential  of  the  training  device  can  be  realized.  The  im¬ 
pact  of  CATCC  on  safe  and  efficient  carrier  recovery  operations  is  suf¬ 
ficient  to  render  these  suggestions  worth  consideration. 
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APPENDIX  C 

OBJECTIVE  PERFORMANCE  MEASURES:  DEFINITIONS 


m 


n 

O' 


The  system  criteria  each  measure  relates  to  and  the  control  position 
or  positions  measured  are  described  in  the  text  (Tables  3  and  4).  Further, 
each  measure  is  generally  defined  in  conjunction  with  the  presentation 
and  discussion  of  data  (Study  Results,  Section  HI).  Some  of  the  measures, 
however,  warrant  a  more  detailed  definition,  or  formulation;  these  are 
presented  below. 


AVERAGE  AIRCRAFT  SEPARATION  ERROR 

A  deviation  of  4-.  5  minutes  from  the  scheduled  separation  time  of 
1.  0  or  2.  0  minutes  between  aircraft  crossing  the  ramp  is  considered 
acceptable.  Unless  there  are  extenuating  circumstances  (e.  g.  ,  degraded 
radar),  any  deviation  larger  than  4\  5  minutes  is  considered  to  reflect  poor 
control.  The  measure  of  average  team  error  on  any  single  recovery,  y, 
was  computed  as  follows: 

i^hi  - 1  51  +  frz  K  ~  5\ 


y~ 


where 


and 


measured 


t^  *  time  interval  between  aircraft^  and  aircraft  j  _  j , 
as  they  cross  the  ramp; 

m  s  number  of  separation  time  intervals  greater  than  1.  5  minutes; 
n  =  number  of  separation  time  intervals  less  than  0.  5  minutes; 
c  s  total  number  of  ramp  crossings. 


COMMUNICATIONS  EFFICIENCY  AND  NOISE  LEVEL 

Both  communications  measures  were  ratings  made  at  the  conclusion 
of  each  recovery  using  a  five  point  continuous  scale: 


Very 

Inefficient 


Very 

Efficient 


Very  High 
Noise  Level 


Very  Low 
Noise  Level 
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RADAR  CONTACT  NOT  ANNOUNCED  TO  PILOT 


Both  Approach  and  Final  controllers  are  required  to  announce  radar 
contact  to  the  pilot  whenever  a  new  contact  is  made.  Four  measures  of 
this  aspect  of  procedural  performance  were  obtained  on  each  recovery, 


one  for  each  Approach  controller  (y.  ,  y_)  and  one  for  each  Final  control- 
lar  (y3,  y4>: 


x  100 


where  N  =  Number  of  contacts  not  announced  by  controller  i, 
na  1 

and  Nm  =  Number  of  contacts  made  by  controller  i. 


NUMBER  OF  CONFLICTIONS 

Jet  aircraft  within  the  recovery  area  (spec.  ,  within  a  50  mile  radius 
of  the  ship)  but  not  on  the  approach  path  are  considered  to  be  in  a  con- 
fliction  situation  if  they  are  not  separated  laterally  by  at  least  3  miles 
or  vertically  by  1000  feet  (a  greater  vertical  separation  is  required  above 
29,000  feet  but  this  was  not  pertinent  to  observed  recoveries).  The  de¬ 
termination  of  which  controller(s)  were  at  fault  was  based  on  observation 
of  the  developing  situation  and  on  later  detailed  analyses  of  the  descriptive 
data  for  that  recovery  (e.  g.  ,  tracing  the  evolution  of  traffic  pattern 
changes  and  relating  them  to  individual  controller  actions). 
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EVALUATION  FORMS 
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CATCC  TRAINER  EVALUATION 

Name _ _  Date 

Primary  Control  Position _ 


INSTRUCTIONS 


As  part  of  an  evaluation  of  training  device  effectiveness  being 
conducted  for  the  U.  S.  Navy  we  would  like  you  to  answer  the  following 
questions  as  completely  and  honestly  as  you  can.  This  is  not  a  test: 
there  are  no  right  or  wrong  answers.  Rather,  your  carefully  considered 


responses  describing  the  role  the  trainer  has  played  in 
for  your  position  are  sought. 


Some  of  the  questions  require  two  kinds  of  answers:  a  rating  of 
the  device  or  its  characteristics  and  your  own  explanation  of  that 
answer.  Please  be  as  detailed  as  you  can. 
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1.  Overall,  how  effective  would  you  say  the  Carrier  Air  Traffic  Control 
Center  training  device  is  as  a  trainer  for  this  position? 


Low 

Effectiveness 


High 

Effectiveness 


2.  How  effective  is  the  trainer  in  developing  the  crew  coordination 
skills  required  as  a  member  of  an  operational  crew? 


Low 

Effectiveness 


High 

Effectiveness 


3.  How  effective  are  the  problems  and  problem  characteristics  in  pro¬ 
viding  training  for  this  position? 


Low 

Effectiveness 


High 

Effectiveness 


Would  you  change  the  problems  presented  to  you,  and,  if  so,  in  what 
way? 
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7.  How  realistic  is  the  response  of  the  training  situation  to  your 

performance  as  compared  to  what  would  happen  in  the  carrier  situation? 


Low 

Realism 

What  changes  would  you  suggest,  if  any? 


High 

Realism 


8.  How  realistic  are  the  training  problems  as  compared  to  the  real 
situation? 


Low 

Realism 


High 

Realism 


Comments? 


9.  How  effective  is  the  symbology  in  aiding  the  controller  at  your 
position? 


Low 

Effectiveness 


High 

Effectiveness 
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What  percentage  of  the  time  did  you  use  the  symbology? 

I...., —I,— —  . i  ...i  . ».ii.  .i.  —I 

o$  25$  50$  75$  ioo$ 


What  characteristics  of  the  device  would  you  say  contribute  to  its 
effectiveness  or  ineffectiveness  (but  have  not  discussed  above)? 
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1?..  Compare  and  rate  the  realism  of  the  trainer  to  the  carrier  situation 
for  the  following  five  major  items: 


Communication. 


Low 

Realism 


High 

Realism 


Guidance  Control 


Low 

Realism 


High 

Realism 


Contingency  Handling  •— «— 

Low 

Realism 


High 

Realism 


Emergency  Situation 


Low 

Realism 


High 
1  Realism 


Your  Performance 


Low 

Realism 


High 

Realism 


13.  Any  additional  comments,  suggestions  or  statements? 


95 


NAVTRADEVCEN  70-C-0258-1 


CONTROLLER  EVALUATION  FORM 


Instruction .  Using  the  three  point  rating  scale  presented  on  the  following 
page,  please  evaluate  and  rate  your  performance  in  the  five  major  areas  of 
communication,  guidance,  emergency  procedures,  contingency  handling,  and 
overall  performance.  Before-  responding,  please  consider  and  assess  your 
performance  relative  to  accuracy  of  information,  adequacy  of  information, 
timeliness  of  infom>*cion,  coordination  of  situation,  safety  adherence  and 
continuance,  requirement  for  information,  and  other  pertinent  standards  you 
feel  should  be  considered  in  the  evaluation  of  your  performance  as  a 
controller. 


Comment.  The  information  gathered  in  this  evaluation  package  will  be  held 
in  the  strictest  confidence  (the  study  evaluators  will  be  the  only  personnel 
to  see  the  evaluation  packages) .  Your  name  and  responses  will  not  be  used 
individually,  but  as  a  collected  whole  in  achieving  an  evaluation  of  con¬ 
troller  performance  in  the  Carrier  Air  Traffic  Control  situation. 


If  there  are  any  questions,  please  ask  them. 


Thank  you  for  your  full  cooperation. 


GUIDANCE 


EMERGENCY 
PROCEDURES 
(if  occur) 


All  previous  exp. 
Prev.  current  cruise 


All  previous  exp. 
Prev.  current  cruise 


CONTINGENCY 

HANDLING 


All  previous  exp. 
Prev.  current  cruise 


OVERALL 

PERFORMANCE 


All  previous  exp. 
Prev.  current  cruise 
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CONTROLLER  EVALUATION 

1.  What  problems  did  you  experience  during  the  exercise? 

2.  Did  the  NTD6  Symbology  poise  any  problems?  Explain. 

3.  Did  you  have  any  coordination  and/or  handling  problems  with  other  CCA 
personnel?  Explain . 

h.  Did  you  maintain  adequate  separation  of  aircraft?  Explain. 

5.  Any  suggestions  or  comments? 


98 


r-V?W1'”  < * 


*  ▼»,*«>*  ~V?'V  i 


NA VTRADEV CEN  70-C-0258-1 


ISO  EVALUATION  OF  EXERCISE 


Instructions.  Please  enter  the  evaluation  information  requested  on  the  follow 
ing  page  based  on  your  total  ISO  experience.  In  responding,  please  consider 
and  assess  the  situation  presently  being  evaluated  versus  a  situation  you 
as  ISO  would  consider  as  a  standard  on  which  to  make  a  comparison  Judgment. 
Realizing  that  each  approach  to  landing  is  unique  in  itself,  it  is  hoped  that 
you  as  ISO  can  through  your  experience  rate  the  performance  of  the  approach 
to  the  overall  approach  situation  being  experienced  on  that  specific  approach. 

Comment ♦  The  information  gathered  in  this  evaluation  package  will  be  held 
in  the  strictest  confidence  (the  study  evaluators  will  he  the  only  personnel 
to  see  the  evaluation  packages).  Your  name  and  responses  will  not  be  used 
individually,  but  as  a  collected  whole  in  achieving  an  evaluation  of  con¬ 
troller  performance  in  the  Carrier  Air  Traffic  Control  situation. 

If  there  are  any  question,  please  ask  them. 

Thank  you  for  your  full  cooperation. 
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LSO  EVALUATION  RATING  FORM 

LSO  Name:  Date:  Term  Time  _ 

Start  Time _ 

Elapsed  Time 


100 
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LSO  Evaluation 

1.  Do  you  fee]  there  is  any  performance  difference  between  Final  Controller  A 
or  Pinal  Controller  B?  Explain. 


2.  Were  the  missed  app  due  to  pilot  technique  or  controller  technique 
(Please  specify  A/C)? 


3.  Any  suggestions  or  comments? 
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PILOT'S  EVALUATION  OP  EXERCISE 


Instructions ■  Using  the  three  point  rating  scale  presented  on  the  following 
page,  please  rate  each  specific  controller's  performance  for  the  five  major 
evaluation  areas  of  communication,  guidance,  emergency  procedures,  contingency 
handling,  and  overall  performance.  Before  responding,  please  consider  and 
assess  the  controller's  performance  relative  to  accuracy  of  information,  ade¬ 
quacy  of  information,  timeliness  of  information,  coordination  of  situation, 
safety  adherance  and  continuous,  requirement  for  information,  and  other 
pertinent  standards  you  feel  should  be  considered  in  the  evaluation  of  con¬ 
troller  performance. 

Comment.  The  information  gathered  in  this  evaluation  package  will  be  held  in 
the  strictest  confidence.  The  study  evaluators  will  be  the  only  personnel  to 
see  the  evaluation  packages.  Your  name  and  responses  will  not  be  used  indi¬ 
vidually,  but  as  a  collected  whole  in  achieving  an  evaluation  of  controller 
performance  in  the  Carrier  Air  Traffic  Control  situation. 


If  there  are  any  question,  please  ask  them. 
Thank  you  for  your  full  cooperation. 


PILOT  EVALUATION  RATING  FORM 
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GLOSSARY 


Bingo 


An  order  to  an  aircraft  to  proceed  immediately  to  a  divert 
field. 


Bolter 


Aircraft  touches  down  but  fails  to  trap. 


Confliction  See  Appendix  3  for  definition. 


Delta 


EAT 


A  six  minute  holding  pattern  at  altitude  and  position 
indicated. 

Expected  Approach  Time.  The  future  time  at  which  an  air¬ 
craft  is  cleared  to  penetrate  from  a  preassigned  fix.  Air¬ 
craft  commence  approach  at  EAT  if  no  further  instructions 
received. 


Go  Around.  Aircraft  completes  approach  but  does  not 
touchdown,  entering  the  bolter /wavcoff  pattern  instead. 

Landing  Signal  Officer.  Provides  aircraft  control  instruc¬ 
tions  from  visual  acquisition  (normally  3/4  to  1  mile  out) 
to  touchdown. 


Mode  III 
Recovery 


Approaches  are  manually  flown  in  response  to  CATCC 
instructions. 


PC&E 

Ramp  Time 

Random 

Recovery 


Operational  Readiness  Inspection. 

Program  Control  and  Evaluation.  The  control  center  for 
the  CATCC  mockup. 

Time  at  which  the  carrier  ramp  is  scheduled  to  be  open 
for  recovery  operations. 


Aircraft  are  directed  to  carrier  for  recovery  when  possible 
upon  arrival,  rather  than  assembling  in  Marshalling  area 
and  commencing  approach  at  EAT. 
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Scheduled 

Recovery 

W/O 


Aircraft  assemble  in  Marshalling  area  at  scheduled  time 
and  depart  on  scheduled  EATs  unless  otherwise  directed. 

Waveoff.  An  instruction  to  continue  flight,  entering  bolter/ 
waveoff  pattern. 
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